{"id":214,"date":"2024-05-02T18:58:45","date_gmt":"2024-05-02T18:58:45","guid":{"rendered":"https:\/\/permutationcity.co.uk\/bp\/?p=214"},"modified":"2024-07-27T08:45:47","modified_gmt":"2024-07-27T08:45:47","slug":"dream-debugging","status":"publish","type":"post","link":"https:\/\/permutationcity.co.uk\/bp\/2024\/05\/02\/dream-debugging\/","title":{"rendered":"Dream debugging"},"content":{"rendered":"\n<p>At the weekend I watched a <a href=\"https:\/\/youtu.be\/72y2EC5fkcE?si=aLTk9Pyi7CtJNBT\">Tomorrow Corporation Tech Demo<\/a>, that&#8217;s the games company that developed <a href=\"https:\/\/en.wikipedia.org\/wiki\/World_of_Goo\">World of Goo<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Little_Inferno\">Little Inferno<\/a>. They show off some tools to help development and debugging. Nice development tools but an amazing debugging tool and I want it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-basics\">The basics<\/h2>\n\n\n\n<p>It has the basics you&#8217;d expect from an integrated editor and debugger:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>See the code.<\/li>\n\n\n\n<li>Run the code.<\/li>\n\n\n\n<li>Set breakpoints.<\/li>\n\n\n\n<li>Step through the code.<\/li>\n\n\n\n<li>Inspect variables.<\/li>\n<\/ul>\n\n\n\n<p>None of that is surprising.\nIt does have a bonus that you don&#8217;t always find:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Hot loading of data and code assets.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-dream\">The dream<\/h2>\n\n\n\n<p>We are use to stepping through code line by line following along what&#8217;s happening.\nSometimes you see the problem as it happens which is great.\nHowever sometimes you miss the problem and the first you know about it is\nan assert, a crash or suddenly the variables looking all wonky.\nIn that case we have to restart then run and \/ or step through the code trying to get back there.\nMaybe it&#8217;s at the beginning of the code but often it&#8217;s not.\nGetting back to the interesting point can be difficult, however careful you were with breakpoints.\nNot in this debugger.\nYou just step <em>backwards<\/em> through the code.\nYou effectively reverse time in the debugger and step back to a previous state.\nThe problem you just missed hasn&#8217;t happened.\nStep back again if you want, as much as you want.\nYou can re-examine the code and step forwards.\nGo back and forth as many times as you like.<\/p>\n\n\n\n<p>Profiling code to find a bottleneck can be awkward.\nYou might need to rebuild or run it in a special mode.\nThe added instrumentation can make things run slowly so getting to the point of interest can take longer.\nIf you&#8217;re lucky you can disable profiling until you get there.\nWhen you&#8217;re there you collect data, stop the program and start trawling through the figures.\nNot in this debugger.\nProfiling is not included by default but it&#8217;s easy to hot load the profiler.\nIn the middle of a normal session you just recompile the full executable and start collecting data,\nno interruptions.\nVisualisations of the data are available immediately and you can drill down to the culprit function.\nNot only can you jump to the function responsible in the editor,\nyou can also jump backwards in time to when this profile data was collected.\nIt lets you investigate what was causing the slowdown in the exact situation where it was slow.\nUpdate the code and continue profiling, see if it&#8217;s fixed.\nWhen you finished you can remove the profiler and continue the debugging session.<\/p>\n\n\n\n<p>If you&#8217;re working in a team bug reports may come in from anyone. Whoever it is bug reports come in many flavours. You might get one that carefully describes the exact steps needed to create the problem. That could be a 15 step process involving specific files, menu items to pick, buttons to press. Repeating all that and recreating the problem can be hard. If you can&#8217;t recreate it are the steps wrong or did you just do them wrong. On the other hand you may just get told that it&#8217;s broken when they loaded a file. Maybe it&#8217;s a specific file, maybe it&#8217;s any file but the bug report doesn&#8217;t say. It works with the first file you try, it works with the second, I guess you need to chase up the reporter. Not with this debugger. Every session saves enough information to completely repeat the session. Every bug report can link to the exact occurrence of the problem. Just click a link, scrub through a timeline of the program&#8217;s execution and see the problem. Debug the problem. See exactly what line, what variable is causing the problem. Update the code and see if it&#8217;s fixed.<\/p>\n\n\n\n<p>I think that all sounds like a dream debugging experience.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-technology\">The technology<\/h2>\n\n\n\n<p>I don&#8217;t know how they&#8217;ve done this and it seems that this video is the main source of information available.\nThe developers have complete control over the tech stack.\nThey have their own programming language, compiler front engine and back engine,\neditor, debugger, build system and game engine which is impressive.\nI don&#8217;t know if such complete control is required for all these features but\nit probably helps smooth off most of the rough spots.\nWith just the video to go on the rest of this section has some supposition on my part.<\/p>\n\n\n\n<p>There are probably a few things going on to make this possible:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We know they save out a session file for each run but it&#8217;s not too big. This needs to include the version of the source and assets that were used. That could be an existing source control version but it must also be able to include local changes or maybe all local changes also go to the server. If someone applies hot fixes during a session those will also have to be stored. All user input would have to be recorded and timestamped to an exact frame.<\/li>\n\n\n\n<li>We know their system is deterministic. That means given the same setup and same inputs it will always proceed in the same way. For this to work <em>everything<\/em> has to be deterministic. If one thing can vary run to run then it can effect everything else. Any random number generation will have to come from the same seed. Rendering probably has to be at a fixed frame rate or operates separately from the rest of the system.<\/li>\n\n\n\n<li>We know systems must be coupled. To use small session files the game engine and debugger must know about source control. To support dynamic graphics during debugging the debugger must know about the engine. There&#8217;s probably a lot of other things going on.<\/li>\n<\/ul>\n\n\n\n<p>All of this isn&#8217;t completely unknown. Searching finds multiple mentions, e.g. <a href=\"https:\/\/www.sourceware.org\/gdb\/news\/reversible.html\">GDB and Reverse Debugging<\/a> and <a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/step-back-while-debugging-with-intellitrace\/\">Step-back while debugging with IntelliTrace<\/a>.<\/p>\n\n\n\n<p>I&#8217;ve even heard of it before in combination with virtual machines. That sounded hard but not <em>too<\/em> hard. Some operations are perfectly reversible, say, adding two numbers but other operations are not, say, dividing two numbers. Once you&#8217;ve divided your numbers some information has just gone. Fortunately you can have complete control over a virtual machine. Any information that would be lost can be stored and retrieved when necessary. Alternatively it would be possible to take periodic snapshots of the machine&#8217;s memory. Then you could fake a single &#8220;step back&#8221; by jumping back to the previous snapshot then quickly stepping forwards to the right instruction. The more snapshots you take the faster the you can fake it but the more data you need to store. Maybe something clever could be done with virtual memory so that you only needed to consider pages that had been changed.<\/p>\n\n\n\n<p>Are they doing this the <em>easy<\/em> way with a virtual machines or are they running on real hardware? No idea.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"in-the-end\">In the end<\/h2>\n\n\n\n<p>While I&#8217;d love to be able to use something like this it I&#8217;m not sure when it will arrive. The GDB and Microsoft articles are both many years old but this video is the first practical application of the idea. It could make a huge difference in day to day productively and I definitely want it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">P.S.<\/h2>\n\n\n\n<p>There is a bit more detail in the <a href=\"https:\/\/tomorrowcorporation.com\/posts\/how-we-make-games-at-tomorrow-corp-our-custom-tools-tech-demo\">comments at Tomorrow Corporation<\/a>,<br>specifically in reply to Joseph Garvin&#8217;s questions. They capture the game state by simply copying the game&#8217;s heap in between frames. This happens every 2 minutes to facilitate the timeline and more fine grained but temporary snapshots taken more frequently. That&#8217;s low level enough that random number generation and input buffers may be captured without additional effort. It makes me think that there is some fraction of the game engine that is outside this full debugging framework but it&#8217;s still sounds great.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>At the weekend I watched a Tomorrow Corporation Tech Demo, that&#8217;s the games company that developed World of Goo and Little Inferno. They show off some tools to help development and debugging. Nice development tools but an amazing debugging tool and I want it. The basics It has the basics you&#8217;d expect from an integrated [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[32],"tags":[12,24],"class_list":["post-214","post","type-post","status-publish","format-standard","hentry","category-dream","tag-debugging","tag-videos"],"_links":{"self":[{"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/posts\/214","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/comments?post=214"}],"version-history":[{"count":3,"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/posts\/214\/revisions"}],"predecessor-version":[{"id":218,"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/posts\/214\/revisions\/218"}],"wp:attachment":[{"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/media?parent=214"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/categories?post=214"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/permutationcity.co.uk\/bp\/wp-json\/wp\/v2\/tags?post=214"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}