Hudson River Trading - Tech Blog

https://rsshub.app/hudsonrivertrading/blog (RSS订阅地址)

Designing a SystemVerilog Language Server

<h2 class="wp-block-heading" id="h-background-hrt-s-hardware-tooling-stack">Background: HRT’s Hardware Tooling Stack</h2> At HRT, pushing technical boundaries often means moving beyond the constraints of standard software. Some of the ways we accomplish this are by deploying FPGA and ASIC solutions which allow us to reach latency and throughput levels unattainable with general-purpose CPUs. We build the custom logic needed for these platforms using <a href="https://en.wikipedia.org/wiki/SystemVerilog">SystemVerilog</a>. The hardware workflow relies on simulators to test the designs and synthesizers to ‘compile’ designs onto hardware. Our workflow uses a mix of open source and proprietary tools. Historically, most of these tools have been costly and proprietary – in the past we even had to pay for software compilers. While open source tooling for software has evolved and matured quite a bit over the last few decades, open source hardware tooling has developed at a slower pace. This has led to many issues: <ul class="wp-block-list"> <li>Limited simulator licences make CI nearly impossible</li> <li>No free and widely used linter, formatter, language server, etc.</li> <li>Different proprietary tools accept vastly different language constructs, resulting in many tool-dependent macros to achieve the same thing</li> </ul> We’re eager to address these issues by adopting, improving upon, and adding to the open source hardware tools. For Verilator, this means bug fixes and improvements to the open source simulator. For Cocotb, this means actively maintaining and improving that testing library, and notably increasing the parity between Verilator and proprietary simulators such that the same tests work on both simulators. Utilizing these tools allows our hardware development flow to behave more like a cohesive part of our overall software team, and also allows us to use the same build and test tools for efficient CI and share testing workflows. It may seem counterintuitive for a trading firm, typically protective of its intellectual property, to contribute to and publish open-source projects. However, we believe it is vital to the long term success of software projects to provide extra validation, opinions, and contributions that move these projects forward. We’re also very happy to lower the barrier of entry for hardware development and improve the lives of hardware developers around the world. Proprietary tools still have their place in our workflow. For example, proprietary simulators tend to have faster incremental builds, which many hardware designers need to utilize to achieve a quick iteration loop. Our workflow that we’ve developed allows hardware designers to simply flip a command line switch to run the same exact tests on either Verilator or a proprietary simulator. This allows Verilator to catch CI regressions quickly by running thousands of tests in parallel, while also allowing for developers to use a proprietary simulator for rapid debugging and testing of a design. <h2 class="wp-block-heading" id="h-what-is-a-language-server">What is a language server?</h2> The project we built and released is a <a href="https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification">language server</a>, which is a tool that provides intelligent code navigation and instant feedback into popular editors like VS Code, Vim, and Emacs for a particular language. It’s a protocol based on the VSCode api, allowing the server to be written in any language, not just typescript. The server is based on the Slang project, which is a SystemVerilog frontend and linter. Slang was created by our own Mike Popoloski, and he’s continued to develop features and maintain that project. According to the <a href="https://github.com/chipsalliance/sv-tests">ChipsAlliance test suite</a>, Slang is the fastest and most compliant SystemVerilog frontend, which in hardware compilers is the engine that parses code, checks for errors, and builds the design hierarchy. <h2 class="wp-block-heading" id="h-why-build-a-language-server">Why build a language server?</h2> As part of my role, I need to read and understand parts of our hardware codebase. It irked me (and others on my team) every time I had to scour our codebase for declarations when seeking to understand a piece of code. I couldn’t even imagine how much time our team would save if we had proper editor features like we did in other languages, where a simple cmd+click would bring you to a symbol definition. I started exploring this space by contributing to the most popular vscode extension at the time, first by adding slang linting. This closed a critical feedback loop in our development: users could now immediately see when they misspelled a variable or missed a semicolon, rather than having to switch to a terminal and fix those after running a build. I continued with contributions to the vscode extension, adding cross-file goto-definition and completion support. However, due to limitations in the underlying parser, I hit a wall in developing new features. In order to progress, we needed to build a proper language server using slang, which would be faster, more accurate, and work on any editor. Luckily, at HRT we have the ability to work on greenfield projects through a program called “Surge”, in which groups of engineers choose to come together to explore and build new projects. Both our FPGA team and the Slang library were first created as Surge projects many years ago. Last year, we created slang-server during Surge, and we’re proud to have recently <a href="https://github.com/hudson-trading/slang-server" type="link" id="https://github.com/hudson-trading/slang-server">open sourced it.</a> In the rest of this article we’re going to explore some technical aspects of the server. First we’re going to explore the constructs that the Slang library provides and how we used those constructs to implement some core language features. Then, we’re going to look at optimizations that were made for the indexer’s latency and memory usage, which is a critical part of the language server that collects workspace-wide symbols when the editor is opened. <h2 class="wp-block-heading">What the Slang Library Provides</h2> Slang works in two main passes: parsing and elaboration. Phase 1: Parsing The first phase of the process is Parsing, composed of Lexing, Preprocessing and Parsing. Lexing breaks the raw text file into individual “words” or tokens. “Trivia” such as comments, whitespace, and tool pragmas are attached to the token that immediately follows them. Next, Preprocessing runs a preprocessor similar to those found in C and C++. This step expands macros, follows ifdef statements, and expands <code>include directives</code>. Finally, the main Parsing step arranges all of these processed tokens into a syntax tree, which is a hierarchical structure of syntax nodes, with each node containing tokens and other syntax nodes. Phase 2: Elaboration  Slang walks through the set of syntax trees starting at a top level module. While doing so it calculates parameter values (e.g., “This RAM has a width of 64 bits”) and connects module instances to one another to form the final design. Since modules can be parameterized like C++ templates, Slang can’t just read the code once; it has to “build” the design to see what it actually looks like, and check for issues like width mismatches in each parameterization. During this process it converts the syntax tree into symbols – abstract objects that represent the definitions of your modules, variables, and functions. When using the <code>slang</code> binary (the linter), it runs this entire process to print <a href="https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#diagnostic">diagnostics</a> (errors and warnings) to your terminal. The <code>slang-server</code>, on the other hand, presents these diagnostics directly to an IDE over the <a href="https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/">Language Server Protocol</a>, providing useful info to users as they type. Here is an illustration of some of the information that we get from Parsing and Elaboration, created with Slang’s <code>--cst-json</code> and <code>--ast-json</code> commands. <div data-wp-context="{ "autoclose": true, "accordionItems": [] }" data-wp-interactive="core/accordion" role="group" class="wp-block-accordion is-layout-flow wp-block-accordion-is-layout-flow"> <div data-wp-class--is-open="state.isOpen" data-wp-context="{ "id": "accordion-item-2", "openByDefault": false }" data-wp-init="callbacks.initAccordionItems" data-wp-on-window--hashchange="callbacks.hashChange" class="wp-block-accordion-item is-layout-flow wp-block-accordion-item-is-layout-flow"> <h3 class="wp-block-accordion-heading has-text-color has-background has-link-color has-medium-font-size wp-elements-d12cd949cd9e648ceb42dc842a57726f" style="color:#151526;background-color:#eeeff8"><button aria-expanded="false" aria-controls="accordion-item-2-panel" data-wp-bind--aria-expanded="state.isOpen" data-wp-on--click="actions.toggle" data-wp-on--keydown="actions.handleKeyDown" id="accordion-item-2" type="button" class="wp-block-accordion-heading__toggle">SystemVerilog Input+</button></h3> <div inert="" aria-labelledby="accordion-item-2" data-wp-bind--inert="!state.isOpen" id="accordion-item-2-panel" role="region" class="wp-block-accordion-panel is-layout-flow wp-block-accordion-panel-is-layout-flow"> <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers verilogcode" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(12px, .75rem, 18px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .75rem);line-height:clamp(16px, 1rem, 24px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><svg xmlns="http://www.w3.org/2000/svg" width="54" height="14" viewBox="0 0 54 14"><g fill="none" fill-rule="evenodd" transform="translate(1 1)"><circle cx="6" cy="6" r="6" fill="#FF5F56" stroke="#E0443E" stroke-width=".5"></circle><circle cx="26" cy="6" r="6" fill="#FFBD2E" stroke="#DEA123" stroke-width=".5"></circle><circle cx="46" cy="6" r="6" fill="#27C93F" stroke="#1AAB29" stroke-width=".5"></circle></g></svg><pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">module Comparator ( input logic [3:0] count, input logic [3:0] target, output logic match ); // always_comb automatically triggers whenever 'count' or 'target' changes always_comb begin if (count == target) match = 1'b1; else match = 1'b0; end endmodule module CounterTop ( input logic clk, input logic rst_n, // Active-low reset input logic [3:0] limit, output logic at_limit ); logic [3:0] current_count; // always_ff describes sequential logic (flip-flops) // It requires a sensitivity list, usually the clock edge always_ff @(posedge clk or negedge rst_n) begin if (!rst_n) begin current_count <= 4'b0000; end else begin current_count <= current_count + 1'b1; end end // Instantiate the combinational module Comparator comp_inst ( .count(current_count), .target(limit), .match(at_limit) ); endmodule</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>module Comparator ( input logic [3:0] count, input logic [3:0] target, output logic match ); // always_comb automatically triggers whenever 'count' or 'target' changes always_comb begin if (count == target) match = 1'b1; else match = 1'b0; end endmodule module CounterTop ( input logic clk, input logic rst_n, // Active-low reset input logic [3:0] limit, output logic at_limit ); logic [3:0] current_count; // always_ff describes sequential logic (flip-flops) // It requires a sensitivity list, usually the clock edge always_ff @(posedge clk or negedge rst_n) begin if (!rst_n) begin current_count <= 4'b0000; end else begin current_count <= current_count + 1'b1; end end // Instantiate the combinational module Comparator comp_inst ( .count(current_count), .target(limit), .match(at_limit) ); endmodule</code></pre></div> </div> </div> </div> <div data-wp-context="{ "autoclose": false, "accordionItems": [] }" data-wp-interactive="core/accordion" role="group" class="wp-block-accordion is-layout-flow wp-block-accordion-is-layout-flow"> <div data-wp-class--is-open="state.isOpen" data-wp-context="{ "id": "accordion-item-3", "openByDefault": false }" data-wp-init="callbacks.initAccordionItems" data-wp-on-window--hashchange="callbacks.hashChange" class="wp-block-accordion-item has-background is-layout-flow wp-block-accordion-item-is-layout-flow" style="background-color:#eeeff8"> <h3 class="wp-block-accordion-heading has-text-color has-background has-link-color has-medium-font-size wp-elements-40dd5e9db03f4c88e121c147bd2b1bd0" style="color:#151526;background-color:#eeeff8"><button aria-expanded="false" aria-controls="accordion-item-3-panel" data-wp-bind--aria-expanded="state.isOpen" data-wp-on--click="actions.toggle" data-wp-on--keydown="actions.handleKeyDown" id="accordion-item-3" type="button" class="wp-block-accordion-heading__toggle">Parse Pass Output: Concrete Syntax Tree+</button></h3> <div inert="" aria-labelledby="accordion-item-3" data-wp-bind--inert="!state.isOpen" id="accordion-item-3-panel" role="region" class="wp-block-accordion-panel is-layout-flow wp-block-accordion-panel-is-layout-flow"> <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers verilogcode" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(12px, .75rem, 18px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(4 * 0.6 * .75rem);line-height:clamp(16px, 1rem, 24px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><svg xmlns="http://www.w3.org/2000/svg" width="54" height="14" viewBox="0 0 54 14"><g fill="none" fill-rule="evenodd" transform="translate(1 1)"><circle cx="6" cy="6" r="6" fill="#FF5F56" stroke="#E0443E" stroke-width=".5"></circle><circle cx="26" cy="6" r="6" fill="#FFBD2E" stroke="#DEA123" stroke-width=".5"></circle><circle cx="46" cy="6" r="6" fill="#27C93F" stroke="#1AAB29" stroke-width=".5"></circle></g></svg><pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">{ "syntaxTrees": [ { "kind": "SyntaxTree", "root": { "kind": "CompilationUnit", "members": [ { "kind": "ModuleDeclaration", "header": { "kind": "ModuleHeader", "moduleKeyword": { "kind": "ModuleKeyword", "text": "module" }, "name": { "kind": "Identifier", "text": "Comparator", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": { "kind": "AnsiPortList", "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": [ { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "count", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "target", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "OutputKeyword", "text": "output", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "match", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } ], "closeParen": { "kind": "CloseParenthesis", "text": ")", "trivia": [ { "kind": "EndOfLine", "text": "\n" } ] } }, "semi": { "kind": "Semicolon", "text": ";" } }, "members": [ { "kind": "AlwaysCombBlock", "keyword": { "kind": "AlwaysCombKeyword", "text": "always_comb", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// always_comb automatically triggers whenever 'count' or 'target' changes" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "statement": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ConditionalStatement", "ifKeyword": { "kind": "IfKeyword", "text": "if", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "predicate": { "kind": "ConditionalPredicate", "conditions": [ { "kind": "ConditionalPattern", "expr": { "kind": "EqualityExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "count" } }, "operatorToken": { "kind": "DoubleEquals", "text": "==", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "target", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } } ] }, "closeParen": { "kind": "CloseParenthesis", "text": ")" }, "statement": { "kind": "ExpressionStatement", "expr": { "kind": "AssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "match", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "Equals", "text": "=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "1", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "1" } } }, "semi": { "kind": "Semicolon", "text": ";" } }, "elseClause": { "kind": "ElseClause", "elseKeyword": { "kind": "ElseKeyword", "text": "else", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "clause": { "kind": "ExpressionStatement", "expr": { "kind": "AssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "match", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "Equals", "text": "=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "1", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "0" } } }, "semi": { "kind": "Semicolon", "text": ";" } } } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } } ], "endmodule": { "kind": "EndModuleKeyword", "text": "endmodule", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" } ] } }, { "kind": "ModuleDeclaration", "header": { "kind": "ModuleHeader", "moduleKeyword": { "kind": "ModuleKeyword", "text": "module", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" } ] }, "name": { "kind": "Identifier", "text": "CounterTop", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": { "kind": "AnsiPortList", "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": [ { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "clk", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "rst_n", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// Active-low reset" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "limit", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "OutputKeyword", "text": "output", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "at_limit", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } ], "closeParen": { "kind": "CloseParenthesis", "text": ")", "trivia": [ { "kind": "EndOfLine", "text": "\n" } ] } }, "semi": { "kind": "Semicolon", "text": ";" } }, "members": [ { "kind": "DataDeclaration", "type": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] }, "declarators": [ { "kind": "Declarator", "name": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } ], "semi": { "kind": "Semicolon", "text": ";" } }, { "kind": "AlwaysFFBlock", "keyword": { "kind": "AlwaysFFKeyword", "text": "always_ff", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// always_ff describes sequential logic (flip-flops)" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// It requires a sensitivity list, usually the clock edge" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "statement": { "kind": "TimingControlStatement", "timingControl": { "kind": "EventControlWithExpression", "at": { "kind": "At", "text": "@", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "expr": { "kind": "ParenthesizedEventExpression", "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "BinaryEventExpression", "left": { "kind": "SignalEventExpression", "edge": { "kind": "PosEdgeKeyword", "text": "posedge" }, "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "clk", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "operatorToken": { "kind": "OrKeyword", "text": "or", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "SignalEventExpression", "edge": { "kind": "NegEdgeKeyword", "text": "negedge", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "rst_n", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } } }, "statement": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ConditionalStatement", "ifKeyword": { "kind": "IfKeyword", "text": "if", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "predicate": { "kind": "ConditionalPredicate", "conditions": [ { "kind": "ConditionalPattern", "expr": { "kind": "UnaryLogicalNotExpression", "operatorToken": { "kind": "Exclamation", "text": "!" }, "operand": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "rst_n" } } } } ] }, "closeParen": { "kind": "CloseParenthesis", "text": ")" }, "statement": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ExpressionStatement", "expr": { "kind": "NonblockingAssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "LessThanEquals", "text": "<=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "4", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "0000" } } }, "semi": { "kind": "Semicolon", "text": ";" } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "elseClause": { "kind": "ElseClause", "elseKeyword": { "kind": "ElseKeyword", "text": "else", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "clause": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ExpressionStatement", "expr": { "kind": "NonblockingAssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "LessThanEquals", "text": "<=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "AddExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "Plus", "text": "+", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "1", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "1" } } } }, "semi": { "kind": "Semicolon", "text": ";" } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } } }, { "kind": "HierarchyInstantiation", "type": { "kind": "Identifier", "text": "Comparator", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// Instantiate the combinational module" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "instances": [ { "kind": "HierarchicalInstance", "decl": { "kind": "InstanceName", "name": { "kind": "Identifier", "text": "comp_inst", "trivia": [ { "kind": "Whitespace", "text": " " } ] } }, "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "connections": [ { "kind": "NamedPortConnection", "dot": { "kind": "Dot", "text": ".", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "name": { "kind": "Identifier", "text": "count" }, "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "SimplePropertyExpr", "expr": { "kind": "SimpleSequenceExpr", "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count" } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } }, { "kind": "Comma", "text": "," }, { "kind": "NamedPortConnection", "dot": { "kind": "Dot", "text": ".", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "name": { "kind": "Identifier", "text": "target" }, "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "SimplePropertyExpr", "expr": { "kind": "SimpleSequenceExpr", "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "limit" } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } }, { "kind": "Comma", "text": "," }, { "kind": "NamedPortConnection", "dot": { "kind": "Dot", "text": ".", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "name": { "kind": "Identifier", "text": "match" }, "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "SimplePropertyExpr", "expr": { "kind": "SimpleSequenceExpr", "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "at_limit" } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } } ], "closeParen": { "kind": "CloseParenthesis", "text": ")", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } ], "semi": { "kind": "Semicolon", "text": ";" } } ], "endmodule": { "kind": "EndModuleKeyword", "text": "endmodule", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" } ] } } ] } } ] }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>{ "syntaxTrees": [ { "kind": "SyntaxTree", "root": { "kind": "CompilationUnit", "members": [ { "kind": "ModuleDeclaration", "header": { "kind": "ModuleHeader", "moduleKeyword": { "kind": "ModuleKeyword", "text": "module" }, "name": { "kind": "Identifier", "text": "Comparator", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": { "kind": "AnsiPortList", "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": [ { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "count", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "target", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "OutputKeyword", "text": "output", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "match", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } ], "closeParen": { "kind": "CloseParenthesis", "text": ")", "trivia": [ { "kind": "EndOfLine", "text": "\n" } ] } }, "semi": { "kind": "Semicolon", "text": ";" } }, "members": [ { "kind": "AlwaysCombBlock", "keyword": { "kind": "AlwaysCombKeyword", "text": "always_comb", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// always_comb automatically triggers whenever 'count' or 'target' changes" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "statement": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ConditionalStatement", "ifKeyword": { "kind": "IfKeyword", "text": "if", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "predicate": { "kind": "ConditionalPredicate", "conditions": [ { "kind": "ConditionalPattern", "expr": { "kind": "EqualityExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "count" } }, "operatorToken": { "kind": "DoubleEquals", "text": "==", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "target", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } } ] }, "closeParen": { "kind": "CloseParenthesis", "text": ")" }, "statement": { "kind": "ExpressionStatement", "expr": { "kind": "AssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "match", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "Equals", "text": "=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "1", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "1" } } }, "semi": { "kind": "Semicolon", "text": ";" } }, "elseClause": { "kind": "ElseClause", "elseKeyword": { "kind": "ElseKeyword", "text": "else", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "clause": { "kind": "ExpressionStatement", "expr": { "kind": "AssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "match", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "Equals", "text": "=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "1", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "0" } } }, "semi": { "kind": "Semicolon", "text": ";" } } } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } } ], "endmodule": { "kind": "EndModuleKeyword", "text": "endmodule", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" } ] } }, { "kind": "ModuleDeclaration", "header": { "kind": "ModuleHeader", "moduleKeyword": { "kind": "ModuleKeyword", "text": "module", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" } ] }, "name": { "kind": "Identifier", "text": "CounterTop", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": { "kind": "AnsiPortList", "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "ports": [ { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "clk", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "rst_n", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "InputKeyword", "text": "input", "trivia": [ { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// Active-low reset" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "limit", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, { "kind": "Comma", "text": "," }, { "kind": "ImplicitAnsiPort", "header": { "kind": "VariablePortHeader", "direction": { "kind": "OutputKeyword", "text": "output", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dataType": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "declarator": { "kind": "Declarator", "name": { "kind": "Identifier", "text": "at_limit", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } ], "closeParen": { "kind": "CloseParenthesis", "text": ")", "trivia": [ { "kind": "EndOfLine", "text": "\n" } ] } }, "semi": { "kind": "Semicolon", "text": ";" } }, "members": [ { "kind": "DataDeclaration", "type": { "kind": "LogicType", "keyword": { "kind": "LogicKeyword", "text": "logic", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "dimensions": [ { "kind": "VariableDimension", "openBracket": { "kind": "OpenBracket", "text": "[", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "specifier": { "kind": "RangeDimensionSpecifier", "selector": { "kind": "SimpleRangeSelect", "left": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "3" } }, "range": { "kind": "Colon", "text": ":" }, "right": { "kind": "IntegerLiteralExpression", "literal": { "kind": "IntegerLiteral", "text": "0" } } } }, "closeBracket": { "kind": "CloseBracket", "text": "]" } } ] }, "declarators": [ { "kind": "Declarator", "name": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } ], "semi": { "kind": "Semicolon", "text": ";" } }, { "kind": "AlwaysFFBlock", "keyword": { "kind": "AlwaysFFKeyword", "text": "always_ff", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// always_ff describes sequential logic (flip-flops)" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// It requires a sensitivity list, usually the clock edge" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "statement": { "kind": "TimingControlStatement", "timingControl": { "kind": "EventControlWithExpression", "at": { "kind": "At", "text": "@", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "expr": { "kind": "ParenthesizedEventExpression", "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "BinaryEventExpression", "left": { "kind": "SignalEventExpression", "edge": { "kind": "PosEdgeKeyword", "text": "posedge" }, "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "clk", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } }, "operatorToken": { "kind": "OrKeyword", "text": "or", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "SignalEventExpression", "edge": { "kind": "NegEdgeKeyword", "text": "negedge", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "rst_n", "trivia": [ { "kind": "Whitespace", "text": " " } ] } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } } }, "statement": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ConditionalStatement", "ifKeyword": { "kind": "IfKeyword", "text": "if", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "predicate": { "kind": "ConditionalPredicate", "conditions": [ { "kind": "ConditionalPattern", "expr": { "kind": "UnaryLogicalNotExpression", "operatorToken": { "kind": "Exclamation", "text": "!" }, "operand": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "rst_n" } } } } ] }, "closeParen": { "kind": "CloseParenthesis", "text": ")" }, "statement": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ExpressionStatement", "expr": { "kind": "NonblockingAssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "LessThanEquals", "text": "<=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "4", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "0000" } } }, "semi": { "kind": "Semicolon", "text": ";" } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "elseClause": { "kind": "ElseClause", "elseKeyword": { "kind": "ElseKeyword", "text": "else", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "clause": { "kind": "SequentialBlockStatement", "begin": { "kind": "BeginKeyword", "text": "begin", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "items": [ { "kind": "ExpressionStatement", "expr": { "kind": "NonblockingAssignmentExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "LessThanEquals", "text": "<=", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "AddExpression", "left": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count", "trivia": [ { "kind": "Whitespace", "text": " " } ] } }, "operatorToken": { "kind": "Plus", "text": "+", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "right": { "kind": "IntegerVectorExpression", "size": { "kind": "IntegerLiteral", "text": "1", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "base": { "kind": "IntegerBase", "text": "'b" }, "value": { "kind": "IntegerLiteral", "text": "1" } } } }, "semi": { "kind": "Semicolon", "text": ";" } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } } } ], "end": { "kind": "EndKeyword", "text": "end", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } } }, { "kind": "HierarchyInstantiation", "type": { "kind": "Identifier", "text": "Comparator", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " }, { "kind": "LineComment", "text": "// Instantiate the combinational module" }, { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "instances": [ { "kind": "HierarchicalInstance", "decl": { "kind": "InstanceName", "name": { "kind": "Identifier", "text": "comp_inst", "trivia": [ { "kind": "Whitespace", "text": " " } ] } }, "openParen": { "kind": "OpenParenthesis", "text": "(", "trivia": [ { "kind": "Whitespace", "text": " " } ] }, "connections": [ { "kind": "NamedPortConnection", "dot": { "kind": "Dot", "text": ".", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "name": { "kind": "Identifier", "text": "count" }, "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "SimplePropertyExpr", "expr": { "kind": "SimpleSequenceExpr", "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "current_count" } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } }, { "kind": "Comma", "text": "," }, { "kind": "NamedPortConnection", "dot": { "kind": "Dot", "text": ".", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "name": { "kind": "Identifier", "text": "target" }, "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "SimplePropertyExpr", "expr": { "kind": "SimpleSequenceExpr", "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "limit" } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } }, { "kind": "Comma", "text": "," }, { "kind": "NamedPortConnection", "dot": { "kind": "Dot", "text": ".", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] }, "name": { "kind": "Identifier", "text": "match" }, "openParen": { "kind": "OpenParenthesis", "text": "(" }, "expr": { "kind": "SimplePropertyExpr", "expr": { "kind": "SimpleSequenceExpr", "expr": { "kind": "IdentifierName", "identifier": { "kind": "Identifier", "text": "at_limit" } } } }, "closeParen": { "kind": "CloseParenthesis", "text": ")" } } ], "closeParen": { "kind": "CloseParenthesis", "text": ")", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "Whitespace", "text": " " } ] } } ], "semi": { "kind": "Semicolon", "text": ";" } } ], "endmodule": { "kind": "EndModuleKeyword", "text": "endmodule", "trivia": [ { "kind": "EndOfLine", "text": "\n" }, { "kind": "EndOfLine", "text": "\n" } ] } } ] } } ] }</code></pre></div> </div> </div> </div> <div data-wp-context="{ "autoclose": false, "accordionItems": [] }" data-wp-interactive="core/accordion" role="group" class="wp-block-accordion is-layout-flow wp-block-accordion-is-layout-flow"> <div data-wp-class--is-open="state.isOpen" data-wp-context="{ "id": "accordion-item-4", "openByDefault": false }" data-wp-init="callbacks.initAccordionItems" data-wp-on-window--hashchange="callbacks.hashChange" class="wp-block-accordion-item is-layout-flow wp-block-accordion-item-is-layout-flow"> <h3 class="wp-block-accordion-heading has-text-color has-background has-link-color has-medium-font-size wp-elements-f92ae795b230391951bcabfcfafa6656" style="color:#151526;background-color:#eeeff8"><button aria-expanded="false" aria-controls="accordion-item-4-panel" data-wp-bind--aria-expanded="state.isOpen" data-wp-on--click="actions.toggle" data-wp-on--keydown="actions.handleKeyDown" id="accordion-item-4" type="button" class="wp-block-accordion-heading__toggle">Elaboration Pass Output: Abstract Syntax Tree+</button></h3> <div inert="" aria-labelledby="accordion-item-4" data-wp-bind--inert="!state.isOpen" id="accordion-item-4-panel" role="region" class="wp-block-accordion-panel is-layout-flow wp-block-accordion-panel-is-layout-flow"> <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers verilogcode" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(12px, .75rem, 18px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(3 * 0.6 * .75rem);line-height:clamp(16px, 1rem, 24px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><svg xmlns="http://www.w3.org/2000/svg" width="54" height="14" viewBox="0 0 54 14"><g fill="none" fill-rule="evenodd" transform="translate(1 1)"><circle cx="6" cy="6" r="6" fill="#FF5F56" stroke="#E0443E" stroke-width=".5"></circle><circle cx="26" cy="6" r="6" fill="#FFBD2E" stroke="#DEA123" stroke-width=".5"></circle><circle cx="46" cy="6" r="6" fill="#27C93F" stroke="#1AAB29" stroke-width=".5"></circle></g></svg><pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">{ "design": { "name": "$root", "kind": "Root", "addr": 6308034860160, "members": [ { "name": "", "kind": "CompilationUnit", "addr": 6308035446328 }, { "name": "CounterTop", "kind": "Instance", "addr": 6308035447208, "body": { "name": "CounterTop", "kind": "InstanceBody", "addr": 6308035446624, "members": [ { "name": "clk", "kind": "Port", "addr": 6308035447352, "type": "logic", "direction": "In", "internalSymbol": "6308035447480 clk" }, { "name": "clk", "kind": "Variable", "addr": 6308035447480, "type": "logic", "lifetime": "Static" }, { "name": "rst_n", "kind": "Port", "addr": 6308035447624, "type": "logic", "direction": "In", "internalSymbol": "6308035447752 rst_n" }, { "name": "rst_n", "kind": "Variable", "addr": 6308035447752, "type": "logic", "lifetime": "Static" }, { "name": "limit", "kind": "Port", "addr": 6308035447896, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035448024 limit" }, { "name": "limit", "kind": "Variable", "addr": 6308035448024, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "at_limit", "kind": "Port", "addr": 6308035448168, "type": "logic", "direction": "Out", "internalSymbol": "6308035448296 at_limit" }, { "name": "at_limit", "kind": "Variable", "addr": 6308035448296, "type": "logic", "lifetime": "Static" }, { "name": "current_count", "kind": "Variable", "addr": 6308035446880, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "", "kind": "ProceduralBlock", "addr": 6308035447024, "procedureKind": "AlwaysFF", "body": { "kind": "Timed", "timing": { "kind": "EventList", "events": [ { "kind": "SignalEvent", "expr": { "kind": "NamedValue", "type": "logic", "symbol": "6308035447480 clk" }, "edge": "PosEdge" }, { "kind": "SignalEvent", "expr": { "kind": "NamedValue", "type": "logic", "symbol": "6308035447752 rst_n" }, "edge": "NegEdge" } ] }, "stmt": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "Conditional", "conditions": [ { "expr": { "kind": "UnaryOp", "type": "logic", "op": "LogicalNot", "operand": { "kind": "NamedValue", "type": "logic", "symbol": "6308035447752 rst_n" } } } ], "check": "None", "ifTrue": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic[3:0]", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" }, "right": { "kind": "Conversion", "type": "logic[3:0]", "operand": { "kind": "IntegerLiteral", "type": "bit[3:0]", "value": "4'b0", "constant": "4'b0" }, "constant": "4'b0" }, "isNonBlocking": true } } }, "ifFalse": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic[3:0]", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" }, "right": { "kind": "BinaryOp", "type": "logic[3:0]", "op": "Add", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" }, "right": { "kind": "Conversion", "type": "logic[3:0]", "operand": { "kind": "IntegerLiteral", "type": "bit[0:0]", "value": "1'b1", "constant": "1'b1" }, "constant": "4'b1" } }, "isNonBlocking": true } } } } } } }, { "name": "comp_inst", "kind": "Instance", "addr": 6308035448472, "body": { "name": "Comparator", "kind": "InstanceBody", "addr": 6308035448608, "members": [ { "name": "count", "kind": "Port", "addr": 6308035451440, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451568 count" }, { "name": "count", "kind": "Variable", "addr": 6308035451568, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "target", "kind": "Port", "addr": 6308035451712, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451840 target" }, { "name": "target", "kind": "Variable", "addr": 6308035451840, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "match", "kind": "Port", "addr": 6308035451984, "type": "logic", "direction": "Out", "internalSymbol": "6308035452112 match" }, { "name": "match", "kind": "Variable", "addr": 6308035452112, "type": "logic", "lifetime": "Static" }, { "name": "", "kind": "ProceduralBlock", "addr": 6308035448920, "procedureKind": "AlwaysComb", "body": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "Conditional", "conditions": [ { "expr": { "kind": "BinaryOp", "type": "logic", "op": "Equality", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035451568 count" }, "right": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035451840 target" } } } ], "check": "None", "ifTrue": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic", "left": { "kind": "NamedValue", "type": "logic", "symbol": "6308035452112 match" }, "right": { "kind": "Conversion", "type": "logic", "operand": { "kind": "IntegerLiteral", "type": "bit[0:0]", "value": "1'b1", "constant": "1'b1" }, "constant": "1'b1" }, "isNonBlocking": false } }, "ifFalse": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic", "left": { "kind": "NamedValue", "type": "logic", "symbol": "6308035452112 match" }, "right": { "kind": "Conversion", "type": "logic", "operand": { "kind": "IntegerLiteral", "type": "bit[0:0]", "value": "1'b0", "constant": "1'b0" }, "constant": "1'b0" }, "isNonBlocking": false } } } } } ], "definition": "6308031980544 Comparator" }, "connections": [ { "port": { "name": "count", "kind": "Port", "addr": 6308035451440, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451568 count" }, "expr": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" } }, { "port": { "name": "target", "kind": "Port", "addr": 6308035451712, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451840 target" }, "expr": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035448024 limit" } }, { "port": { "name": "match", "kind": "Port", "addr": 6308035451984, "type": "logic", "direction": "Out", "internalSymbol": "6308035452112 match" }, "expr": { "kind": "Assignment", "type": "logic", "left": { "kind": "NamedValue", "type": "logic", "symbol": "6308035448296 at_limit" }, "right": { "kind": "EmptyArgument", "type": "logic" }, "isNonBlocking": false } } ] } ], "definition": "6308031981312 CounterTop" }, "connections": [ ] } ] }, "definitions": [ { "name": "Comparator", "kind": "Definition", "addr": 6308031980544, "defaultNetType": "6308034857920 wire", "definitionKind": "Module", "defaultLifetime": "Static", "unconnectedDrive": "None", "cellDefine": false }, { "name": "CounterTop", "kind": "Definition", "addr": 6308031981312, "defaultNetType": "6308034857920 wire", "definitionKind": "Module", "defaultLifetime": "Static", "unconnectedDrive": "None", "cellDefine": false } ] }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>{ "design": { "name": "$root", "kind": "Root", "addr": 6308034860160, "members": [ { "name": "", "kind": "CompilationUnit", "addr": 6308035446328 }, { "name": "CounterTop", "kind": "Instance", "addr": 6308035447208, "body": { "name": "CounterTop", "kind": "InstanceBody", "addr": 6308035446624, "members": [ { "name": "clk", "kind": "Port", "addr": 6308035447352, "type": "logic", "direction": "In", "internalSymbol": "6308035447480 clk" }, { "name": "clk", "kind": "Variable", "addr": 6308035447480, "type": "logic", "lifetime": "Static" }, { "name": "rst_n", "kind": "Port", "addr": 6308035447624, "type": "logic", "direction": "In", "internalSymbol": "6308035447752 rst_n" }, { "name": "rst_n", "kind": "Variable", "addr": 6308035447752, "type": "logic", "lifetime": "Static" }, { "name": "limit", "kind": "Port", "addr": 6308035447896, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035448024 limit" }, { "name": "limit", "kind": "Variable", "addr": 6308035448024, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "at_limit", "kind": "Port", "addr": 6308035448168, "type": "logic", "direction": "Out", "internalSymbol": "6308035448296 at_limit" }, { "name": "at_limit", "kind": "Variable", "addr": 6308035448296, "type": "logic", "lifetime": "Static" }, { "name": "current_count", "kind": "Variable", "addr": 6308035446880, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "", "kind": "ProceduralBlock", "addr": 6308035447024, "procedureKind": "AlwaysFF", "body": { "kind": "Timed", "timing": { "kind": "EventList", "events": [ { "kind": "SignalEvent", "expr": { "kind": "NamedValue", "type": "logic", "symbol": "6308035447480 clk" }, "edge": "PosEdge" }, { "kind": "SignalEvent", "expr": { "kind": "NamedValue", "type": "logic", "symbol": "6308035447752 rst_n" }, "edge": "NegEdge" } ] }, "stmt": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "Conditional", "conditions": [ { "expr": { "kind": "UnaryOp", "type": "logic", "op": "LogicalNot", "operand": { "kind": "NamedValue", "type": "logic", "symbol": "6308035447752 rst_n" } } } ], "check": "None", "ifTrue": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic[3:0]", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" }, "right": { "kind": "Conversion", "type": "logic[3:0]", "operand": { "kind": "IntegerLiteral", "type": "bit[3:0]", "value": "4'b0", "constant": "4'b0" }, "constant": "4'b0" }, "isNonBlocking": true } } }, "ifFalse": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic[3:0]", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" }, "right": { "kind": "BinaryOp", "type": "logic[3:0]", "op": "Add", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" }, "right": { "kind": "Conversion", "type": "logic[3:0]", "operand": { "kind": "IntegerLiteral", "type": "bit[0:0]", "value": "1'b1", "constant": "1'b1" }, "constant": "4'b1" } }, "isNonBlocking": true } } } } } } }, { "name": "comp_inst", "kind": "Instance", "addr": 6308035448472, "body": { "name": "Comparator", "kind": "InstanceBody", "addr": 6308035448608, "members": [ { "name": "count", "kind": "Port", "addr": 6308035451440, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451568 count" }, { "name": "count", "kind": "Variable", "addr": 6308035451568, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "target", "kind": "Port", "addr": 6308035451712, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451840 target" }, { "name": "target", "kind": "Variable", "addr": 6308035451840, "type": "logic[3:0]", "lifetime": "Static" }, { "name": "match", "kind": "Port", "addr": 6308035451984, "type": "logic", "direction": "Out", "internalSymbol": "6308035452112 match" }, { "name": "match", "kind": "Variable", "addr": 6308035452112, "type": "logic", "lifetime": "Static" }, { "name": "", "kind": "ProceduralBlock", "addr": 6308035448920, "procedureKind": "AlwaysComb", "body": { "kind": "Block", "blockKind": "Sequential", "body": { "kind": "Conditional", "conditions": [ { "expr": { "kind": "BinaryOp", "type": "logic", "op": "Equality", "left": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035451568 count" }, "right": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035451840 target" } } } ], "check": "None", "ifTrue": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic", "left": { "kind": "NamedValue", "type": "logic", "symbol": "6308035452112 match" }, "right": { "kind": "Conversion", "type": "logic", "operand": { "kind": "IntegerLiteral", "type": "bit[0:0]", "value": "1'b1", "constant": "1'b1" }, "constant": "1'b1" }, "isNonBlocking": false } }, "ifFalse": { "kind": "ExpressionStatement", "expr": { "kind": "Assignment", "type": "logic", "left": { "kind": "NamedValue", "type": "logic", "symbol": "6308035452112 match" }, "right": { "kind": "Conversion", "type": "logic", "operand": { "kind": "IntegerLiteral", "type": "bit[0:0]", "value": "1'b0", "constant": "1'b0" }, "constant": "1'b0" }, "isNonBlocking": false } } } } } ], "definition": "6308031980544 Comparator" }, "connections": [ { "port": { "name": "count", "kind": "Port", "addr": 6308035451440, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451568 count" }, "expr": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035446880 current_count" } }, { "port": { "name": "target", "kind": "Port", "addr": 6308035451712, "type": "logic[3:0]", "direction": "In", "internalSymbol": "6308035451840 target" }, "expr": { "kind": "NamedValue", "type": "logic[3:0]", "symbol": "6308035448024 limit" } }, { "port": { "name": "match", "kind": "Port", "addr": 6308035451984, "type": "logic", "direction": "Out", "internalSymbol": "6308035452112 match" }, "expr": { "kind": "Assignment", "type": "logic", "left": { "kind": "NamedValue", "type": "logic", "symbol": "6308035448296 at_limit" }, "right": { "kind": "EmptyArgument", "type": "logic" }, "isNonBlocking": false } } ] } ], "definition": "6308031981312 CounterTop" }, "connections": [ ] } ] }, "definitions": [ { "name": "Comparator", "kind": "Definition", "addr": 6308031980544, "defaultNetType": "6308034857920 wire", "definitionKind": "Module", "defaultLifetime": "Static", "unconnectedDrive": "None", "cellDefine": false }, { "name": "CounterTop", "kind": "Definition", "addr": 6308031981312, "defaultNetType": "6308034857920 wire", "definitionKind": "Module", "defaultLifetime": "Static", "unconnectedDrive": "None", "cellDefine": false } ] }</code></pre></div> </div> </div> </div> <h2 class="wp-block-heading">Implementing Goto Definition</h2> Now let’s explore how we use this information to implement one of the most commonly used features that a language server provides: <a href="https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_definition">Goto Definition</a>. A compiler like Slang will produce symbols that reference lower and lower levels of abstraction; however, we need pointers going the other way. For example, Slang has a <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/ast/Compilation.h">Compilation</a>, which points at <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/ast/Symbol.h">Symbols</a>, which point at <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/syntax/SyntaxNode.h">SyntaxNodes</a>, which point at <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/parsing/Token.h">Tokens</a>, which finally point at <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/text/SourceLocation.h">SourceLocations</a>. When asking the language server for a definition, we need to traverse this chain in the opposite direction using what’s called a reverse index. Let’s explore how this is accomplished at each step. SourceLocation (offset) -> Token: Binary Search Using a binary search was chosen instead of array or hashmap for reduced memory and better startup latency when a file is opened. Token -> Syntax: Token index While visiting the syntax tree, we create a map of tokens to their parent syntax, since tokens in Slang don’t have parent pointers (Syntax Nodes do). Some sort of binary search on the tree itself will not work since the tree’s tokens may have come from macro expansions, so comparing those source locations would not make sense. Token/Syntax -> Symbol: If we’re looking at an expression: <ul class="wp-block-list"> <li>We can use Slang’s <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/ast/Lookup.h">lookup</a> functionality if we have a NameSyntax and a Scope object. A NameSyntax is a class of syntax nodes that look like a name, and may include things like array accesses and struct member accesses. To get this we can follow parent pointers up to a name syntax. To grab a <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/ast/Scope.h">Scope</a> object, we create a map of syntax nodes to scopes on the initial indexing of the shallow compilation.</li> </ul> If we’re on a symbol declaration: <ul class="wp-block-list"> <li>We can visit the symbols in our file when it’s opened and index the name token of each symbol. This is used to access the symbol for go-to references and hovers, since we’re already at the symbol declaration.</li> </ul> If we’re using named ports/params/args/struct fields: <ul class="wp-block-list"> <li>We can either index on startup, or crawl back to the symbol definition and get the arguments from there.</li> </ul> There are plenty of other cases to think about that we won’t go into here, such as import/export statements, macro arguments, macro usages, etc. Once we have the symbol, we can get the symbol’s syntax to use in both hovers and goto-defs, and also display relevant information like type information, bitwidth, and more from the symbol. Shallow Compilations The “Compilation” or AST that we use only needs to provide the symbols visible in the current document, so we can use what we call a Shallow Compilation to only elaborate what we need. Most hardware tools typically require that you compile and elaborate an entire design, but we’re able to tweak Slang to provide just the information needed. The shallow compilation only adds syntax trees of directly referenced modules before elaborating. This limited depth means we can recompile the shallow compilation on every keystroke and keep up with the user, rather than performing a costly elaboration on an entire design. This has the added benefit that the user isn’t required to select a design to get language features, which some other SystemVerilog language servers require. Features for analyzing an entire design do exist in the server, but even when a design is set, the server uses a shallow compilation to keep up with the user’s typing and provide core language features on every keystroke. <h2 class="wp-block-heading">Optimizing the Indexer</h2> Now let’s explore a critical part of the language server: the indexer. This costly process runs when the language server is first started. It crawls the workspace for SystemVerilog files and makes mappings of things like module names to files that contain those modules. It’s crucial that this indexing process runs in a reasonable time so that the user isn’t left waiting when opening up their editor. Let’s explore some optimizations that were made in the indexer. Optimization #1: Avoid shared memory and lock contention This code looks fairly innocent: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">void populateIndexForSingleFile(const fs::path& path, Indexer::IndexedPath& dest) { dest.path = path.string(); if (auto tree = slang::syntax::SyntaxTree::fromFile(path.string())) { getDataFromTree(tree->get(), dest.relevantSymbols, dest.relevantMacros); } else { ERROR("Error parsing file for indexing: {}", path.string()); return; } };</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>void populateIndexForSingleFile(const fs::path& path, Indexer::IndexedPath& dest) { dest.path = path.string(); if (auto tree = slang::syntax::SyntaxTree::fromFile(path.string())) { getDataFromTree(tree->get(), dest.relevantSymbols, dest.relevantMacros); } else { ERROR("Error parsing file for indexing: {}", path.string()); return; } };</code></pre></div> However, it’s actually very inefficient if you explore what it does in more depth. The <code>SyntaxTree::fromFile</code> method was added for convenience in testing. It uses a global source manager, which is the class that manages the file data read from disk. For efficiency, this class ensures that each file is only read once. Also, to enable parallel parsing, this SourceManager has locks on most of its methods. These are useful optimizations during normal usage in the slang linter, but in our use case these are hindrances since each indexing operation is completely independent. There’s no time saved by locking and checking if the file was already parsed, since each file is indexed exactly once and we don’t bother expanding included files during indexing. Realizing this, we can instead use one SourceManager per thread on the stack to greatly speed up our indexing. Before, each thread had to reach up into a shared cache level to grab this source manager and contend for the lock. Now each thread has no contention and no cache invalidation from sharing a source manager. Optimization #2: std::unordered_map<SmallVector<>> instead of std::multimap<> It’s possible that a top level symbol name is declared in multiple files – for example, different packages of the same name that hold different parameterizations of a design, or different versions of a module. Because of this, a multimap was initially chosen to store the module to file mapping since it can store a range of values for a key. However, it has log(n) lookup. This cost is also incurred while building the index. An unordered map is a better choice for performance, but if we store a vector in the value, that’s another malloc that will often be just one file. Slang has the exact utility for this: a <a href="https://github.com/MikePopoloski/slang/blob/master/include/slang/util/SmallVector.h">SmallVector</a> implementation. This stores a fixed amount of elements directly in the object before deferring to a normal vector, which saves some indirection. Slang and other compilers use this construct extensively to reduce extra memory allocations when the size of a list is expected to be small. Optimization #3: Only store paths once File paths can take up a lot of memory in our data structures, since the same path will be found in different maps. Also, hashing this string can be expensive. By <a href="https://en.wikipedia.org/wiki/Interning_(computer_science)">interning</a> it, or associating it with an integer id, we hash and store the string once and reuse that hash. Optimization #4: Crawl workspace instead of globbing Globs were initially used to specify which files to index; however, globs are expensive to compute and to crawl, and it’s tricky to specify the four common file extensions that SystemVerilog code lives in. To speed this up and simplify usage, we can crawl the directory directly instead, and allow the user to specify directories that it should skip over – for example, directories containing build artifacts or software. These are real optimizations that were added before adding the goto-references feature, resulting in about a 10x speedup. After adding the referenced top level symbols to the indexing process, it was still about a 4x improvement, providing a meaningful impact for the users of our codebase. We tested it on <a href="https://github.com/lowRISC/opentitan" type="link" id="https://github.com/lowRISC/opentitan">the largest open source SystemVerilog repo</a>, which comes in at 45MB of code and is composed of 3955 files. This was indexed in .41 seconds on an M1 Macbook Pro, composed of .13s crawling and .28s parsing and indexing. <h2 class="wp-block-heading" id="h-wrapping-up">Wrapping Up</h2> HRT chooses to invest in developer efficiency and experience, and we’re proud to share these projects with the hardware community. It is this exact commitment to developer efficiency that has enabled us to scale up and trade on over 200 markets around the world while maintaining a headcount of only around 1,150 people. If you’re a hardware designer or verification engineer looking to work on cutting-edge silicon in a modern development environment, check out our hardware roles <a href="https://www.hudsonrivertrading.com/careers/?job-category=hardware-engineering%2C">here</a>. Or, if you’re interested in working on similar challenging problems close to modern or custom hardware, check out our software roles <a href="https://www.hudsonrivertrading.com/careers/?job-category=software-engineeringc%2Cparent_software-engineeringc%2Csoftware-engineeringpython%2C">here</a>. To try out the language server, simply install it on any VSCode based editor. There’s no longer a need to build it from source; the vscode client will now prompt to auto-install the latest binary from the releases page. Instructions for other editors can be found <a href="https://hudson-trading.github.io/slang-server/start/installing/#neovim">here</a>. This year during Surge, members of our FPGA team and I are making slang-format, a SystemVerilog formatter. If you’re interested in updates like the addition of the formatter, follow along on the <a href="https://github.com/hudson-trading/slang-server">slang-server repo</a>. Sources <ol class="wp-block-list"> <li><a href="https://github.com/MikePopoloski/slang">https://github.com/MikePopoloski/slang</a> </li> <li><a href="https://github.com/hudson-trading/slang-server">https://github.com/hudson-trading/slang-server</a> </li> <li><a href="http://ithare.com/infographics-operation-costs-in-cpu-clock-cycles/">http://ithare.com/infographics-operation-costs-in-cpu-clock-cycles/</a></li> </ol> Recommended Reading <ul class="wp-block-list"> <li><a href="https://en.algorithmica.org/hpc/">https://en.algorithmica.org/hpc/</a></li> </ul>

2026/3/3

HRTWorker: A SharedWorker Framework for Viewing Real-Time Trading

<h2 class="wp-block-heading hrtbeatentrybot" id="h-fullstack-hrt">Fullstack @ HRT</h2> <blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"> <h3 class="wp-block-heading" id="h-at-hudson-river-trading-our-fullstack-team-bridges-the-gap-between-people-and-our-highly-automated-systems-we-build-the-complex-interfaces-and-tools-that-let-people-navigate-control-and-extend-hrt-s-vast-trading-infrastructure-as-well-as-the-data-it-produces-running-systems-at-this-scale-comes-with-no-shortage-of-engineering-challenges-this-post-explores-one-of-those-challenges-and-how-we-solved-it" style="font-style:normal;font-weight:400">At Hudson River Trading, our Fullstack team bridges the gap between people and our highly automated systems. We build the complex interfaces and tools that let people navigate, control, and extend HRT’s vast trading infrastructure, as well as the data it produces. Running systems at this scale comes with no shortage of engineering challenges. This post explores one of those challenges and how we solved it.</h3> </blockquote> We have built a suite of internal web applications that offer a real-time window into our trading environment. These tools provide our Run The Day (RTD) team and other teams throughout the company the ability to observe and respond to the dynamic intraday state of our trading operations. This suite, known collectively as the ‘Ops GUIs’, handle a large volume of complex structured data which updates at sub-second intervals. The data ranges from process statuses to market rejects to health check results. Alongside these fixed GUIs, we offer a WYSIWYG dashboard builder to support the varied data and application needs of different teams. These dashboards are composed of ‘widgets’ that typically summarize the information available in a full application. Users often have their custom dashboard and some fixed GUIs open at the same time, and each GUI instance can load a non-trivial amount of data into memory. How do we avoid duplicate loading and processing of data without overhauling our frontend stack (e.g. to try and implement a server-side framework or to change our trading systems to be frontend-aware)? Since the advent of the <a href="https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker">SharedWorker</a> interface in 2010, which enables a background thread to be shared across tabs, many web programmers consider this a solved problem. In theory, this is an ideal solution, but in practice, SharedWorkers introduce several development challenges: <ol class="wp-block-list"> <li class="hrtbeatbulletspacer">The only way to communicate with a SharedWorker is via a <a href="https://developer.mozilla.org/en-US/docs/Web/API/MessageChannel">MessageChannel</a>; this is a major deviation from the typical function calls we use in frontend code, especially for async functions or those that return multiple responses (as you’d expect from a streaming API).</li> <li class="hrtbeatbulletspacer">MessageChannels utilize the structured clone algorithm [<a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm">1</a>] which mean objects lose their prototype chain along with some other niche properties when passed between contexts.</li> <li class="hrtbeatbulletspacer">Since SharedWorkers aren’t globally available across all browsers, implementing a fallback for environments without SharedWorker support becomes complex and error-prone.</li> </ol> We designed our solution to allow developers to implement their business logic within a single ‘Worker’ class. This class is instantiated in two ways (boolean passed to the constructor): as a ‘Client’ in the GUI or as an ‘Oracle’ in a SharedWorker (and that’s “client” as in “client / server”, where the “Oracle” is like the “server”). The HRTWorker, depending on whether it’s in client or oracle-mode, handles all underlying communication logic, abstracting it away from the developer. In client-mode, HRTWorker proxies requests to a corresponding SharedWorker with an equivalent oracle-mode instance. Essentially, developers only need to extend ‘HRTWorker’ and implement their business logic. Here is an example of a Worker that a developer might implement: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">class MyWorker extends HRTWorker { constructor(isOracleMode: boolean) { super(isOracleMode) } @remoteExecute async *processData(input: DataSet): AsyncGenerator<Progress, Result> { yield { stage: 'parsing', percent: 0 } const parsed = await this.parse(input) yield { stage: 'analyzing', percent: 50 } const analyzed = await this.analyze(parsed) yield { stage: 'complete', percent: 100 } return analyzed } }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>class MyWorker extends HRTWorker { constructor(isOracleMode: boolean) { super(isOracleMode) } @remoteExecute async *processData(input: DataSet): AsyncGenerator<Progress, Result> { yield { stage: 'parsing', percent: 0 } const parsed = await this.parse(input) yield { stage: 'analyzing', percent: 50 } const analyzed = await this.analyze(parsed) yield { stage: 'complete', percent: 100 } return analyzed } }</code></pre></div> And to make use of it inside of a GUI: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">// In the interest of brevity, we'll abstract away the need to pass a reference to the SharedWorker to the client-mode instances const sharedWorker = new SharedWorker("myworker.js") // false here because it's running in client-mode (i.e. in a web app) const processor = new MyWorker(false) for await (const progress of processor.processData(myData)) { console.log(`${progress.stage}: ${progress.percent}%`) }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>// In the interest of brevity, we'll abstract away the need to pass a reference to the SharedWorker to the client-mode instances const sharedWorker = new SharedWorker("myworker.js") // false here because it's running in client-mode (i.e. in a web app) const processor = new MyWorker(false) for await (const progress of processor.processData(myData)) { console.log(`${progress.stage}: ${progress.percent}%`) }</code></pre></div> And the SharedWorker module: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">// true because it's running in oracle-mode (i.e. in a SharedWorker) new MyWorker(true) // On construction, the HRTWorker (when running in oracle-mode) will hook into the SharedWorkerContext // and start listening for messages so there's no need for us to do anything else here.</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>// true because it's running in oracle-mode (i.e. in a SharedWorker) new MyWorker(true) // On construction, the HRTWorker (when running in oracle-mode) will hook into the SharedWorkerContext // and start listening for messages so there's no need for us to do anything else here.</code></pre></div> Now that we’ve taken a look at how this pattern can be used, let’s dive into how HRTWorker is implemented. There are three significant components to the HRTWorker class: <ol class="wp-block-list"> <li>Connection Management</li> <li class="hrtbeatbulletspacer">Execution Protocol (‘Conversations’)</li> <li>Transparent Method Decoration</li> </ol> Let’s examine each component in detail. <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop" id="h-connection-management">Connection Management</h2> MessagePorts are fundamental to any SharedWorker implementation; they are the connection between different browser contexts. A web application receives a port upon which it can talk to the SharedWorker and vice versa. The first thing that we need to be able to do is to manage the lifecycle of these MessagePorts. The code below illustrates how the SharedWorker (oracle-mode) tracks the originating client for a message and can then reply to that client.  <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">abstract class ConnectionManagerMixin { private ports: Map<string, MessagePort> = new Map() constructor(protected isOracleMode: boolean, protected port?: MessagePort) { // The first example of HRTWorker behaving differently based on the // mode it is instantiated in. In oracle-mode, we need to listen to the // SharedWorkerContext onconnect events to detect when a client connects. if (this.isOracleMode) { self.onconnect = (event: MessageEvent) => { this.registerChannel(event.ports[0]) } } else { if (!this.port) throw new Error("Client-mode requires a port") this.port.onmessage = this.handleMessageFromOracle.bind(this) this.port.start() } } protected registerChannel(port: MessagePort): string { // We'll want an ID to identify clients in logging const id = generateId() // Start the listener for this port port.onmessage = (message) => { // We'll implement this in our Execution Protocol section this.handleMessageFromClient(id, message); } this.ports.set(id, port) // Just a simple example of a welcome message, in reality we might want // to send metadata about the client-mode instance (web app name, user etc.) this.sendMessage(id, { type: 'welcome', id }) return id } protected sendMessage(clientId: string, message: any): void { this.ports.get(clientId)?.postMessage(message) } }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>abstract class ConnectionManagerMixin { private ports: Map<string, MessagePort> = new Map() constructor(protected isOracleMode: boolean, protected port?: MessagePort) { // The first example of HRTWorker behaving differently based on the // mode it is instantiated in. In oracle-mode, we need to listen to the // SharedWorkerContext onconnect events to detect when a client connects. if (this.isOracleMode) { self.onconnect = (event: MessageEvent) => { this.registerChannel(event.ports[0]) } } else { if (!this.port) throw new Error("Client-mode requires a port") this.port.onmessage = this.handleMessageFromOracle.bind(this) this.port.start() } } protected registerChannel(port: MessagePort): string { // We'll want an ID to identify clients in logging const id = generateId() // Start the listener for this port port.onmessage = (message) => { // We'll implement this in our Execution Protocol section this.handleMessageFromClient(id, message); } this.ports.set(id, port) // Just a simple example of a welcome message, in reality we might want // to send metadata about the client-mode instance (web app name, user etc.) this.sendMessage(id, { type: 'welcome', id }) return id } protected sendMessage(clientId: string, message: any): void { this.ports.get(clientId)?.postMessage(message) } }</code></pre></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop" id="h-execution-protocol-conversations">Execution Protocol (‘Conversations’)</h2> Cross-context interactions are modeled as structured ‘conversations’ taking place over a multiplexed channel (the MessagePort for the SharedWorker). <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">interface Conversation { id: string method: string isInitiator: boolean stream: AsyncGenerator cleanup: (() => void)[] } enum MessageType { CALL = 'call', RESPONSE = 'response', COMPLETE = 'complete', ERROR = 'error' CANCEL = 'cancel' }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>interface Conversation { id: string method: string isInitiator: boolean stream: AsyncGenerator cleanup: (() => void)[] } enum MessageType { CALL = 'call', RESPONSE = 'response', COMPLETE = 'complete', ERROR = 'error' CANCEL = 'cancel' }</code></pre></div> A typical conversation/execution follows this pattern: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">Client Oracle | | | CALL(id, method, args) | |-------------------------->| | | (executes method) | | | RESPONSE(id, data1) | |<--------------------------| | RESPONSE(id, data2) | |<--------------------------| | | | COMPLETE(id, finalResult) | |<--------------------------| | | done done</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>Client Oracle | | | CALL(id, method, args) | |-------------------------->| | | (executes method) | | | RESPONSE(id, data1) | |<--------------------------| | RESPONSE(id, data2) | |<--------------------------| | | | COMPLETE(id, finalResult) | |<--------------------------| | | done done</code></pre></div> The oracle-mode HRTWorker automatically handles method execution. When a <code>CALL</code> message is received on the MessagePort by the SharedWorker it: <ol class="wp-block-list"> <li>Looks up the requested method on itself</li> <li>Executes it with the provided arguments</li> <li>Streams any yielded values as <code>RESPONSE</code> messages</li> <li>Sends the final return value as <code>COMPLETE</code></li> <li>Propagates any errors as <code>ERROR</code> messages</li> </ol> In HRTWorker, this looks something like: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">/** * This mixin provides the conversation management framework for when the * HRTWorker is in oracle-mode. * * When instantiated in client-mode, none of the code here will ever be run * (handleMessageFromClient is only called in oracle-mode per our ConnectionManagerMixin) */ abstract class OracleConversationManagerMixin { // All of the active conversations private conversations = new Set<string>(); /** * Called onmessage by the ConnectionManagerMixin */ protected async handleMessageFromClient(clientID: string, event: MessageEvent) { // Note that we expect/need conversationID to be UUIDs generated by the client const { type, method, args, conversationID } = event.data switch (type) { case MessageType.CALL: this.conversations.add(conversationID) await this.executeMethod(clientID, method, args) break case MessageType.CANCEL: this.conversations.delete(conversationID) break } } private async executeMethod(conversationId: string, methodName: string, args: any[]) { try { // Get the actual method to execute const method = this[methodName as keyof this] as Function if (!method) throw new Error(`Method ${methodName} not found`) // Execute and handle streaming results const result = method.apply(this, args) if (result && typeof result.next === 'function') { // Handle AsyncGenerator for await (const value of result) { if (!this.conversations.has(conversationId)) return; // Stop if conversation was cancelled this.sendResponse(conversationId, value) } this.sendComplete(conversationId) } else { // Handle regular return value this.sendComplete(conversationId, await result) } } catch (error) { this.sendError(conversationId, error) } } // Some helpful wrappers to create structured messages to send to the client private sendResponse(id: string, data: any) { this.sendMessage(id, { type: MessageType.RESPONSE, data }) } private sendComplete(id: string, data?: any) { this.sendMessage(id, { type: MessageType.COMPLETE, data }) } private sendError(id: string, error: Error) { this.sendMessage(id, { type: MessageType.ERROR, error: { message: error.message, stack: error.stack, name: error.constructor.name } }) } }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>/** * This mixin provides the conversation management framework for when the * HRTWorker is in oracle-mode. * * When instantiated in client-mode, none of the code here will ever be run * (handleMessageFromClient is only called in oracle-mode per our ConnectionManagerMixin) */ abstract class OracleConversationManagerMixin { // All of the active conversations private conversations = new Set<string>(); /** * Called onmessage by the ConnectionManagerMixin */ protected async handleMessageFromClient(clientID: string, event: MessageEvent) { // Note that we expect/need conversationID to be UUIDs generated by the client const { type, method, args, conversationID } = event.data switch (type) { case MessageType.CALL: this.conversations.add(conversationID) await this.executeMethod(clientID, method, args) break case MessageType.CANCEL: this.conversations.delete(conversationID) break } } private async executeMethod(conversationId: string, methodName: string, args: any[]) { try { // Get the actual method to execute const method = this[methodName as keyof this] as Function if (!method) throw new Error(`Method ${methodName} not found`) // Execute and handle streaming results const result = method.apply(this, args) if (result && typeof result.next === 'function') { // Handle AsyncGenerator for await (const value of result) { if (!this.conversations.has(conversationId)) return; // Stop if conversation was cancelled this.sendResponse(conversationId, value) } this.sendComplete(conversationId) } else { // Handle regular return value this.sendComplete(conversationId, await result) } } catch (error) { this.sendError(conversationId, error) } } // Some helpful wrappers to create structured messages to send to the client private sendResponse(id: string, data: any) { this.sendMessage(id, { type: MessageType.RESPONSE, data }) } private sendComplete(id: string, data?: any) { this.sendMessage(id, { type: MessageType.COMPLETE, data }) } private sendError(id: string, error: Error) { this.sendMessage(id, { type: MessageType.ERROR, error: { message: error.message, stack: error.stack, name: error.constructor.name } }) } }</code></pre></div> However, this implementation still requires manual construction of <code>CALL</code> messages in the web app for each remote invocation. This boilerplate code detracts from the developer experience and introduces room for hard-to-spot errors, which brings us to the final component. <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop" id="h-transparent-method-decoration">Transparent Method Decoration</h2> The final piece of the puzzle uses<a href="https://www.typescriptlang.org/tsconfig/#experimentalDecorators"> typescript experimental decorators</a> – which allow us to wrap a method and access descriptor properties of the method that we’re wrapping – to create a transparent interface: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">function remoteExecute(target: any, methodName: string, descriptor: PropertyDescriptor) { const originalMethod = descriptor.value descriptor.value = function(...args: any[]) { // If we're the executor (oracle), run locally if (this.isOracleMode) { return originalMethod.apply(this, args) } // If we're the client, initiate remote call return this.call(methodName, args) } }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>function remoteExecute(target: any, methodName: string, descriptor: PropertyDescriptor) { const originalMethod = descriptor.value descriptor.value = function(...args: any[]) { // If we're the executor (oracle), run locally if (this.isOracleMode) { return originalMethod.apply(this, args) } // If we're the client, initiate remote call return this.call(methodName, args) } }</code></pre></div> Finally, we can implement a conversation management framework for client-mode which @remoteExecute can invoke. In the below example we can construct an abstraction over the underlying channel (and the remote oracle’s async generator) using promises that can be resolved when we receive <code>RESPONSE</code> messages from our remote oracle. Note that MessagePorts have limitations on the objects that they can serialise between browsing contexts, [<a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm">1</a>] and in this public example we haven’t added any safeguards to detect/warn developers that their methods yield/return unserialisable types. The resulting ClientConversationManagerMixin looks something like this, and since we can control our remoteExecute decorator (to disable the entire remote oracle system if we want), we can seamlessly enable and disable workers depending on whether they’re available or even just via a configuration option. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">TypeScript<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">/** * This mixin for HRTWorker provides client-mode instances the ability to * handle conversations with a SharedWorker. The 'call' method is invoked * by the remoteExecute decorator. */ abstract class ClientConversationManagerMixin { private conversations = new Map<string, any>() /** * Called by the remoteExecute decorator in client-mode to execute a method * with some arguments on the SharedWorker. */ async *call(method: string, args: any[]): AsyncGenerator { // Generate a UUID const conversationID = generateId() // Create conversation tracking let resolveNext: (value: any) => void let rejectNext: (error: Error) => void const conversation = { nextPromise: new Promise((resolve, reject) => { resolveNext = resolve rejectNext = reject }), resolveNext, rejectNext, } // We need to keep track of the resolve/reject functions for our promise // since we'll be receiving responses/errors asynchronously via // our MessagePort's onmessage (which calls handleMessageFromOracle) this.conversations.set(conversationID, conversation) // Send the call this.messagePort.postMessage({ type: 'CALL', conversationID, method, args, }) try { while (true) { const message = await conversation.nextPromise switch (message.type) { // Method has yielded something via its asyncgenerator on the // SharedWorker case 'RESPONSE': // Set up next promise, so we don't miss messages conversation.nextPromise = new Promise((resolve, reject) => { conversation.resolveNext = resolve conversation.rejectNext = reject }) // Yield to our web application yield message.data break // Method has returned a value on the SharedWorker case 'COMPLETE': return message.data // Method has thrown an error on the SharedWorker case 'ERROR': throw new Error(message.error.message) // Something has broken in our protocol default: throw new Error(`Unknown message type: ${message.type}`) } } } finally { this.conversations.delete(conversationID) } } protected cancel(conversationID: string) { const conversation = this.conversations.get(conversationID) if (conversation) { this.messagePort.postMessage({ type: 'CANCEL', conversationID, }) conversation.rejectNext(new Error('Conversation cancelled')) this.conversations.delete(conversationID) } } private handleMessageFromOracle(event: MessageEvent) { const { conversationID, ...message } = event.data const conversation = this.conversations.get(conversationID) if (conversation) { conversation.resolveNext(message) } } }</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>/** * This mixin for HRTWorker provides client-mode instances the ability to * handle conversations with a SharedWorker. The 'call' method is invoked * by the remoteExecute decorator. */ abstract class ClientConversationManagerMixin { private conversations = new Map<string, any>() /** * Called by the remoteExecute decorator in client-mode to execute a method * with some arguments on the SharedWorker. */ async *call(method: string, args: any[]): AsyncGenerator { // Generate a UUID const conversationID = generateId() // Create conversation tracking let resolveNext: (value: any) => void let rejectNext: (error: Error) => void const conversation = { nextPromise: new Promise((resolve, reject) => { resolveNext = resolve rejectNext = reject }), resolveNext, rejectNext, } // We need to keep track of the resolve/reject functions for our promise // since we'll be receiving responses/errors asynchronously via // our MessagePort's onmessage (which calls handleMessageFromOracle) this.conversations.set(conversationID, conversation) // Send the call this.messagePort.postMessage({ type: 'CALL', conversationID, method, args, }) try { while (true) { const message = await conversation.nextPromise switch (message.type) { // Method has yielded something via its asyncgenerator on the // SharedWorker case 'RESPONSE': // Set up next promise, so we don't miss messages conversation.nextPromise = new Promise((resolve, reject) => { conversation.resolveNext = resolve conversation.rejectNext = reject }) // Yield to our web application yield message.data break // Method has returned a value on the SharedWorker case 'COMPLETE': return message.data // Method has thrown an error on the SharedWorker case 'ERROR': throw new Error(message.error.message) // Something has broken in our protocol default: throw new Error(`Unknown message type: ${message.type}`) } } } finally { this.conversations.delete(conversationID) } } protected cancel(conversationID: string) { const conversation = this.conversations.get(conversationID) if (conversation) { this.messagePort.postMessage({ type: 'CANCEL', conversationID, }) conversation.rejectNext(new Error('Conversation cancelled')) this.conversations.delete(conversationID) } } private handleMessageFromOracle(event: MessageEvent) { const { conversationID, ...message } = event.data const conversation = this.conversations.get(conversationID) if (conversation) { conversation.resolveNext(message) } } }</code></pre></div> By combining these components, HRTWorker provides a powerful, reusable way to manage complex communication between browser contexts. Naturally, we’ve only scratched the surface here. If you’re the kind of engineer who loves building robust solutions to challenging problems like this, our Fullstack team at HRT is hiring. Check out our job post <a href="https://www.hudsonrivertrading.com/careers/?search=fullstack">here</a> for more info.

2025/11/24

HeraclesQL: A Python DSL for Writing Alerts

<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"> <h3 class="wp-block-heading" id="h-i-m-on-the-systems-dev-team-at-hrt-my-team-is-responsible-for-software-development-that-supports-systems-network-and-datacenter-engineers-who-maintain-the-computers-hrt-relies-on-for-trading-and-research-while-my-team-has-a-large-variety-of-responsibilities-i-m-focused-on-developing-tools-which-improve-monitoring-and-alerting-in-this-post-i-ll-be-doing-a-deep-dive-on-my-team-s-newly-open-sourced-project-heraclesql" style="font-style:normal;font-weight:400">I’m on the Systems Dev team at HRT. My team is responsible for software development that supports Systems, Network, and Datacenter Engineers who maintain the computers HRT relies on for trading and research. While my team has a large variety of responsibilities, I’m focused on developing tools which improve monitoring and alerting. In this post, I’ll be doing a deep dive on my team’s newly open-sourced project: HeraclesQL.</h3> </blockquote> At HRT, <a href="https://victoriametrics.com/">VictoriaMetrics</a> is an important part of our monitoring stack. It centers around a timeseries database that stores metrics about services and devices. VictoriaMetrics has its own query language for fetching timeseries data called MetricsQL. Over time, we’ve written thousands of alerts using MetricsQL queries. While we love VictoriaMetrics and MetricsQL, we’ve encountered some pain points with MetricsQL as the number of teams using VictoriaMetrics has grown. These break down into three categories: excessive code duplication, poor expressiveness in complex queries, and subtle footguns. HeraclesQL is designed to solve these problems while preserving what makes MetricsQL great. It is a Domain Specific Language (DSL) for writing MetricsQL queries in Python. If you’re familiar with SQLAlchemy for SQL, it can be thought of as filling a similar role. Under the hood, HeraclesQL Python code is essentially a type-safe builder which generates MetricsQL. <a href="https://pypi.org/project/heracles-ql/">We’ve provided it as a Python package on PyPi</a> and <a href="https://github.com/hudson-trading/heracles-ql">made it available on GitHub as open source software</a> under the MIT license. Some features include: <ul class="wp-block-list hrtbeatbulletspacer"> <li class="hrtbeatbulletspacer">MetricsQL-like syntax — HeraclesQL will be immediately familiar to anyone who’s written MetricsQL or PromQL.</li> <li class="hrtbeatbulletspacer">Custom Functions, Parameterizable Expressions, and Variables — No more copy and pasting behavior between alerts. </li> <li class="hrtbeatbulletspacer">Static Type Safety — MyPy and your editor will catch common problems before they occur.</li> <li class="hrtbeatbulletspacer">Meta-alerts — Generate alerts about your alerts to avoid common pitfalls.</li> </ul> Before we do our deep dive, let’s briefly discuss MetricsQL. <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-a-brief-intro-to-metricsql">A Brief Intro to MetricsQL</h2> MetricsQL is the language for querying the <a href="https://victoriametrics.com/">VictoriaMetrics timeseries database</a>. It’s largely based on PromQL, which is the query language for the <a href="https://prometheus.io/">Prometheus timeseries database</a>. MetricsQL and PromQL are extremely similar, so if you know one you essentially know the other. Both VictoriaMetrics and Prometheus are monitoring platforms, which means they typically store metrics about services or devices.  Data is stored in the form of numerically-valued timeseries. Each timeseries is identified by a set of string key-value pairs called labels. Queries use string matching operations to select timeseries based on the value of labels. The result of a query is a vector that contains zero or more timeseries.  Let’s look at some examples: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">A simple MetricsQL query <pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">my_cool_metric{job="my_scrape_job"}</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>my_cool_metric{job="my_scrape_job"}</code></pre></div> This query selects a vector of timeseries with the name <code>my_cool_metric</code> and the label ‘job=”my_scrape_job”. The part of the query inside the curly braces are called matchers. There’s no way to know the number of timeseries this query will select. MetricsQL supports many built-in functions and operators. For example, a query can use the <code>sum</code> function to calculate a timeseries which is the sum of all timeseries matching a selector: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">MetricsQL aggregation<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">sum(http_request_duration{method="POST"})</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>sum(http_request_duration{method="POST"})</code></pre></div> Since VictoriaMetrics is a timeseries database, many functions are designed to calculate values over time. For example, the <code>rate</code> function can calculate a timeseries which contains the growth rate of a timeseries over a specified lookback: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">MetricsQL rollup<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">increase(http_request_count_total{path=~"/api/v2/.*"}[5m])</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>increase(http_request_count_total{path=~"/api/v2/.*"}[5m])</code></pre></div> There’s too much behavior to detail in this post, so I’ll stop here. The<a href="https://docs.victoriametrics.com/victoriametrics/metricsql/"> VictoriaMetrics docs are a great resource</a> for learning more about MetricsQL.  The primary purpose of MetricsQL is to write alerts which notify users of anomalous conditions in a service or device. To this end, VictoriaMetrics supports configured alerting rules. Here’s an example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">VictoriaMetrics alerting rule<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">alert: MyUsefulAlert expr: my_cool_metric{job="my_scrape_job"} > 10 labels: some_cool_label: with_a_useful_value</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>alert: MyUsefulAlert expr: my_cool_metric{job="my_scrape_job"} > 10 labels: some_cool_label: with_a_useful_value</code></pre></div> An alert will be generated if the query defined in the <code>expr</code> field returns any timeseries. One alert will be generated per timeseries. <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-heraclesql">HeraclesQL</h2> The HeraclesQL Python package provides the core DSL, config generation for VictoriaMetrics, and some useful utilities like embedded testing and development tools. We want HeraclesQL to be immediately familiar to anybody who’s written PromQL or MetricsQL. To achieve that, we’ve designed HeraclesQL to feel like writing MetricsQL. It’s easier to see what I mean by looking at our example of a simple MetricsQL query: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">A simple MetricsQL query<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">my_cool_metric{job="my_scrape_job"}</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>my_cool_metric{job="my_scrape_job"}</code></pre></div> Now, here’s what it looks like in HeraclesQL: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Simple MetricsQL query in HeraclesQL <pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">v.my_cool_metric(job="my_scrape_job")</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>v.my_cool_metric(job="my_scrape_job")</code></pre></div> HeraclesQL supports all of MetricsQL’s operators. For example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">MetricsQL query with binary operators<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">(v.left_metric{foo="bar"} + on(hostname) v.right_metric) and v.some_other_metric</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>(v.left_metric{foo="bar"} + on(hostname) v.right_metric) and v.some_other_metric</code></pre></div> Can be written in HeraclesQL like this: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Binary operators in HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">(v.left_metric(foo="bar") + v.right_metric).on("hostname") & v.some_other_metric</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>(v.left_metric(foo="bar") + v.right_metric).on("hostname") & v.some_other_metric</code></pre></div> Heracles’ API is completely MyPy and Pyright compliant – Both your editor and MyPy can warn you about typing issues statically: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Poorly typed HeraclesQL expression<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly=""># example.py ql.rate(v.my_metric(foo="bar"))</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code># example.py ql.rate(v.my_metric(foo="bar"))</code></pre></div> <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">$ mypy example.py example.py:78: error: Argument 1 to "rate" has incompatible type "SelectedInstantVector"; expected "RangeVector" [arg-type]</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ mypy example.py example.py:78: error: Argument 1 to "rate" has incompatible type "SelectedInstantVector"; expected "RangeVector" [arg-type]</code></pre></div> Static typing is important to HeraclesQL’s design because we want HeraclesQL to offer an experience that’s even better than writing MetricsQL. Getting warnings from your editor makes it easier to iterate on queries without resorting to trial and error. <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-how-heraclesql-improves-on-metricsql">How HeraclesQL Improves on MetricsQL</h2> We designed HeraclesQL to address a number of persistent problems we’ve had with writing alerts for Prometheus and VictoriaMetrics. These fall into three categories: <ol class="wp-block-list hrtbeatbulletspacer"> <li class="hrtbeatbulletspacer">Code Duplication</li> <li class="hrtbeatbulletspacer">Expressiveness</li> <li class="hrtbeatbulletspacer">Footguns</li> </ol> Let’s take a look at each one in detail. <h3 class="wp-block-heading hrtbeatentrytop">Code Duplication</h3> Code duplication was the biggest motivation for HeraclesQL. At HRT, we have thousands of alerts across dozens of teams. Nothing can be shared between alerts when using plain MetricsQL — if two teams have alerts on disk usage with different thresholds, they both would need to write almost identical alerts. Before developing HeraclesQL, we considered a few options to reduce duplication. These were all some form of text-templating, and they all suffered from similar problems. Namely, text-templating treats the query as text, not a structured language. This means that templates have to be written very carefully to avoid introducing errors on misuse. Text templating also becomes quite unwieldy for complex templates.  HeraclesQL allows type-safe templating inside expressions. Imagine that we want to write an alert that will fire when disk exhaustion is imminent. We might write something like this: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Root filesystem 10% remaining<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">predict_linear( ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} )[12h:], 1h ) < 0.10</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>predict_linear( ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} )[12h:], 1h ) < 0.10</code></pre></div> This expression returns a non-empty vector if the linear interpolation of the fraction of available filesystem space in one hour is less than 10%. But what if we want another alert that fires when we’re minutes away from exhaustion? With MetricsQL, our only option is to write another alert with a nearly duplicate expression: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Root filesystem 1% remaining<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">predict_linear( ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} )[1h:], 15m ) < 0.01</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>predict_linear( ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} )[1h:], 15m ) < 0.01</code></pre></div> This is a simple case, but this can cause a huge amount of code duplication for more complicated alerts.  In HeraclesQL, we can use a function to avoid duplication: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Disk space running out in HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql v = ql.Selector() def disk_space_running_out( lookback: ql.Duration, extrapolate: ql.Duration, threshold: float, ) -> ql.InstantVector: return ( ql.predict_linear( ( v.node_filesystem_avail_bytes(mountpoint="/") / v.node_filesystem_size_bytes(mountpoint="/") )[lookback:], extrapolate, ) < threshold ) # first alert expression disk_space_running_out(12 * ql.Hour, 1 * ql.Hour, 0.10) # second alert expression disk_space_running_out(1 * ql.Hour, 15 * ql.Minute, 0.01)</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql v = ql.Selector() def disk_space_running_out( lookback: ql.Duration, extrapolate: ql.Duration, threshold: float, ) -> ql.InstantVector: return ( ql.predict_linear( ( v.node_filesystem_avail_bytes(mountpoint="/") / v.node_filesystem_size_bytes(mountpoint="/") )[lookback:], extrapolate, ) < threshold ) # first alert expression disk_space_running_out(12 * ql.Hour, 1 * ql.Hour, 0.10) # second alert expression disk_space_running_out(1 * ql.Hour, 15 * ql.Minute, 0.01)</code></pre></div> Now we can make as many variants of this expression as we want without duplicating any code! HeraclesQL’s internal representation of an expression is essentially the abstract syntax tree of the corresponding MetricsQL expression. The HeraclesQL value is not used to query VictoriaMetrics directly. Instead, it’s used to generate MetricsQL. For example, if we wrote: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Rendering a HeraclesQL query<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">print(disk_space_running_out(12 * ql.Hour, 1h * ql.Hour, 0.10).render())</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>print(disk_space_running_out(12 * ql.Hour, 1h * ql.Hour, 0.10).render())</code></pre></div>  We’d print out MetricsQL: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">MetricsQL generated from HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">predict_linear( ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} )[12h:], 3600000 ) < 0.10</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>predict_linear( ( node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} )[12h:], 3600000 ) < 0.10</code></pre></div> Generating MetricsQL is useful because it allows Heracles code to generate native VictoriaMetrics rule configuration files! <h3 class="wp-block-heading hrtbeatentrytop">Expressiveness</h3> While MetricsQL is concise for simple queries, it can easily become difficult to understand. Even using ‘WITH’ templating, there’s not much flexibility to define variables or avoid repetition. Here’s an alert to use as an example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">A complicated MetricsQL query<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">- alert: APIPoorThroughput expr: |- ( ( rate(http_request_duration_sum{path=~"/api/v1/.*"}[5m]) / rate(http_request_duration_count_total[5m]) ) / changes_prometheus( ( increase(http_response_size_bytes_sum[5m]) / increase(http_response_size_bytes_count_total[5m]) ) * 1M ) ) > 0.005</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>- alert: APIPoorThroughput expr: |- ( ( rate(http_request_duration_sum{path=~"/api/v1/.*"}[5m]) / rate(http_request_duration_count_total[5m]) ) / changes_prometheus( ( increase(http_response_size_bytes_sum[5m]) / increase(http_response_size_bytes_count_total[5m]) ) * 1M ) ) > 0.005</code></pre></div> This alert calculates the 5m average request duration per megabyte of response size. Everybody has their own threshold for readability, but this alert is beginning to reach mine. I can understand it by reading it closely, but it’s difficult to grok right away. Since HeraclesQL queries are just Python code, it’s possible to use variables, branches, loops, and functions to make the query’s intent more clear. We can rewrite this alert using HeraclesQL’s configuration package: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">APIPoorThroughput alert in HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() @rules.alert() def api_poor_throughput() -> config.Alert: lookback = 5 * ql.Minute # define a variable which calculates just the request duration average_request_duration = ql.increase( v.http_request_duration_sum(path=ql.RE("/api/v1/.*"))[lookback] ) / ql.increase(v.http_request_duration_count_total[lookback]) # separately, calculate average response size average_response_size = ( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_response_size_bytes_count_total[lookback]) ) * (1000 * 1000) # use the two variables above to calculate the request duration in terms of response size request_duration_per_response_size = ( average_request_duration / average_response_size ) # finally, return an alert which tests the request_duration_per_response_size against the threshold return config.SimpleAlert(expr=request_duration_per_response_size > 0.005)</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() @rules.alert() def api_poor_throughput() -> config.Alert: lookback = 5 * ql.Minute # define a variable which calculates just the request duration average_request_duration = ql.increase( v.http_request_duration_sum(path=ql.RE("/api/v1/.*"))[lookback] ) / ql.increase(v.http_request_duration_count_total[lookback]) # separately, calculate average response size average_response_size = ( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_response_size_bytes_count_total[lookback]) ) * (1000 * 1000) # use the two variables above to calculate the request duration in terms of response size request_duration_per_response_size = ( average_request_duration / average_response_size ) # finally, return an alert which tests the request_duration_per_response_size against the threshold return config.SimpleAlert(expr=request_duration_per_response_size > 0.005)</code></pre></div> The HeraclesQL version of this alert has several advantages: <ul class="wp-block-list hrtbeatbulletspacer"> <li class="hrtbeatbulletspacer">lookback is a variable so it doesn’t have to be repeated multiple times</li> <li class="hrtbeatbulletspacer">average_request_duration and average_response_size can be calculated separately and stored in named variables</li> <li class="hrtbeatbulletspacer ">The final alert’s expression clearly explains what it does to a reader without needing any comments</li> </ul> With HeraclesQL, we can go even further than this – what if we want different thresholds for a few different paths? HeraclesQL’s config library lets us use parameterizable functions to register the same alert multiple times: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Defining multiple HeraclesQL alerts from one function<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.selector() @rules.alert( "HandlerALowThroughput", path_prefix="/api/v1/handler_a", threshold = 0.005, ) @rules.alert( "HandlerBLowThroughput", path_prefix="/api/v1/handler_b", threshold = 0.0001, ) def api_poor_throughput( path_prefix: str, threshold: float, lookback: ql.Duration = 5*ql.Minute, ) -> config.Alert: # define a variable which calculates just the request duration average_request_duration = ql.increase( v.http_request_duration_sum(path=ql.RE(f"{path_prefix}.*"))[lookback] ) / ql.increase(v.http_request_duration_count_total[lookback]) # separately, calculate average response size average_response_size = ( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_response_size_bytes_count_total[lookback]) ) * (1000 * 1000) # use the two variables above to calculate the request duration in terms of # response size request_duration_per_response_size = ( average_request_duration / average_response_size ) # finally, return an alert which tests the request_duration_per_response_size against the threshold return config.SimpleAlert( expr=request_duration_per_response_size > threshold, )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.selector() @rules.alert( "HandlerALowThroughput", path_prefix="/api/v1/handler_a", threshold = 0.005, ) @rules.alert( "HandlerBLowThroughput", path_prefix="/api/v1/handler_b", threshold = 0.0001, ) def api_poor_throughput( path_prefix: str, threshold: float, lookback: ql.Duration = 5*ql.Minute, ) -> config.Alert: # define a variable which calculates just the request duration average_request_duration = ql.increase( v.http_request_duration_sum(path=ql.RE(f"{path_prefix}.*"))[lookback] ) / ql.increase(v.http_request_duration_count_total[lookback]) # separately, calculate average response size average_response_size = ( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_response_size_bytes_count_total[lookback]) ) * (1000 * 1000) # use the two variables above to calculate the request duration in terms of # response size request_duration_per_response_size = ( average_request_duration / average_response_size ) # finally, return an alert which tests the request_duration_per_response_size against the threshold return config.SimpleAlert( expr=request_duration_per_response_size > threshold, )</code></pre></div> Named variables can also be defined outside of rule functions and referenced multiple times. By default, this will replicate that variable’s sub-expression wherever it is referenced. Often, it’s desirable to create a recording rule from this kind of shared expression. In HeraclesQL, recording rules can be defined using a similar API to alerts, but this can often be overkill for simple cases. Thus, HeraclesQL provides a less verbose way to wrap an expression to produce a recording rule: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">A simple recording rule in HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() lookback: ql.Duration = 5 * ql.Minute # wrap our expression in rules.record to let heracles know that this should # produce a recording rule. The resulting variable is a normal InstantVector average_request_duration = rules.record( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_request_duration_count_total[lookback]), "my_service:avg_request_duration:5m", ) @rules.alert( "HandlerALowThroughput", path_prefix="/api/v1/handler_a", threshold=0.005, ) @rules.alert( "HandlerBLowThroughput", path_prefix="/api/v1/handler_b", threshold=0.0001, ) def api_poor_throughput( path_prefix: str, threshold: float, ) -> config.Alert: average_response_size = ( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_response_size_bytes_count_total[lookback]) ) * (1000 * 1000) # use the average_request_duration variable. When we generate MetricsQL, # this variable will render as a reference to # my_service:avg_request_duration:5m request_duration_per_response_size = ( average_request_duration(path=ql.RE(f"{path_prefix}.*")) / average_response_size ) return config.SimpleAlert( expr=request_duration_per_response_size > threshold, ) @rules.alert() def api_request_duration_very_long() -> config.Alert: return config.SimpleAlert(expr=average_request_duration > 10)</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() lookback: ql.Duration = 5 * ql.Minute # wrap our expression in rules.record to let heracles know that this should # produce a recording rule. The resulting variable is a normal InstantVector average_request_duration = rules.record( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_request_duration_count_total[lookback]), "my_service:avg_request_duration:5m", ) @rules.alert( "HandlerALowThroughput", path_prefix="/api/v1/handler_a", threshold=0.005, ) @rules.alert( "HandlerBLowThroughput", path_prefix="/api/v1/handler_b", threshold=0.0001, ) def api_poor_throughput( path_prefix: str, threshold: float, ) -> config.Alert: average_response_size = ( ql.increase(v.http_request_duration_sum[lookback]) / ql.increase(v.http_response_size_bytes_count_total[lookback]) ) * (1000 * 1000) # use the average_request_duration variable. When we generate MetricsQL, # this variable will render as a reference to # my_service:avg_request_duration:5m request_duration_per_response_size = ( average_request_duration(path=ql.RE(f"{path_prefix}.*")) / average_response_size ) return config.SimpleAlert( expr=request_duration_per_response_size > threshold, ) @rules.alert() def api_request_duration_very_long() -> config.Alert: return config.SimpleAlert(expr=average_request_duration > 10)</code></pre></div> In both rules, <code>average_request_duration</code> will render as a reference to the recording rule.  <h3 class="wp-block-heading hrtbeatentrytop">Footguns</h3> MetricsQL is extremely powerful, but with that power comes complexity. There’s a great deal of subtle behavior which isn’t always obvious. In our experience, the most common mistakes involve unexpectedly missing timeseries. Take a simple expression, for example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">A potentially broken MetricsQL alert<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">increase( my_service_background_job_started_count{job_name="very-important"}[10m] ) < 1</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>increase( my_service_background_job_started_count{job_name="very-important"}[10m] ) < 1</code></pre></div> If my <code>very-important</code> job doesn’t run at least once in a 10m span, this will generate a vector with at least one timeseries and raise an alert. But what happens if <code>my_service_background_job_started_count</code> isn’t in VictoriaMetrics? In that case, the expression will always return an empty vector because the selector <code>my_service_background_job_started_count{job_name="very-important"}</code> will return nothing. I will not get any alerts even though my job definitely isn’t running! This kind of bug can easily occur for metrics which are registered dynamically. If <code>my_service</code> registers new <code>job_name</code> values only when a job starts, this metric would never be generated until the job runs for the first time. To avoid this, it’s best practice to add an absence check to selectors that reference timeseries that may be dynamically produced. However, this is easily overlooked. In practice, we found that alerts at HRT rarely handled timeseries absence correctly.  HeraclesQL provides a way to automatically add this check: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">AlertForMissingData context in HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() # use config.AlertForMissingData to generating a meta-alert that verifies # all the timeseries we expect to select are actually selected with rules.context(config.AlertForMissingData()): @rules.alert() def very_important_job_not_running() -> config.Alert: return config.SimpleAlert( # we don't need to include an absence checks in our expression, # config.AlertForMissingData will do it for us expr=ql.increase( v.my_service_background_job_started_count(job_name="very-important")[ 10 * ql.Minute ] ) > 0, )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() # use config.AlertForMissingData to generating a meta-alert that verifies # all the timeseries we expect to select are actually selected with rules.context(config.AlertForMissingData()): @rules.alert() def very_important_job_not_running() -> config.Alert: return config.SimpleAlert( # we don't need to include an absence checks in our expression, # config.AlertForMissingData will do it for us expr=ql.increase( v.my_service_background_job_started_count(job_name="very-important")[ 10 * ql.Minute ] ) > 0, )</code></pre></div> Before understanding what <code>AlertForMissingData</code> does, we need to take a small detour to explain what <code>rules.context</code> is doing. In addition to registering alerts, the <code>RuleBundle</code> object is able to perform post-processing. All HeraclesQL vector objects store the entire AST of the query, so post-processors are able to walk the AST and generate meta-queries. The context manager above is used to register <code>AlertForMissingData</code> as a post processor for every alert defined within the context. To understand what this does, it’s easiest to look at the output HeraclesQL’s config generation produces: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Generated output<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly=""> rules: - alert: VeryImportantJobNotRunning expr: |- increase( my_service_background_job_started_count{job_name="very-important"}[10m] ) > 0.0 - alert: VeryImportantJobNotRunningDataMissing expr: absent(my_service_background_job_started_count{job_name="very-important"})</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code> rules: - alert: VeryImportantJobNotRunning expr: |- increase( my_service_background_job_started_count{job_name="very-important"}[10m] ) > 0.0 - alert: VeryImportantJobNotRunningDataMissing expr: absent(my_service_background_job_started_count{job_name="very-important"})</code></pre></div> In addition to our alert, Heracles has generated a <code>VeryImportJobNotRunningDataMissing</code> alert which fires when <code>my_service_background_job_started_count</code> is absent. AlertForDataMissing generated this alert by traversing the AST of our first alert’s expression and generating an absence check for each root-level vector selector. Not all common footguns are so straightforward. For example, in MetricsQL, binary operations inherently perform setwise operations. Something as simple as this can have multiple meanings: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">A simple binary operation<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">vector_x + vector_y</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>vector_x + vector_y</code></pre></div> The ‘<code>+</code>’ operator can be thought of as working in two steps: First, it finds all the timeseries in <code>vector_y</code> which have matching labels to timeseries in <code>vector_x</code>, keeping only the intersection. Next, it adds the values of the remaining timeseries to produce a new vector. The net result is an operation which can mean either “add” or “filter” depending on the context. When writing a query, it’s easy to accidentally filter when you only meant to add (in fact, HeraclesQL’s standard library implements generic joins using the <code>+</code> operator). For these ambiguous cases, HeraclesQL provides ‘annotations’ which can be attached to queries to signal intent. In this case, we can write something like this: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Annotated binary operation<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config from heracles.ql import assertions v = ql.Selector() (v.vector_x + v.vector_y).annotate(assertions.no_cardinality_change)</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config from heracles.ql import assertions v = ql.Selector() (v.vector_x + v.vector_y).annotate(assertions.no_cardinality_change)</code></pre></div> When used with the <code>AlertForAssertions</code> context, this will generate a meta-alert which fires when the assertion is not true: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Generated invalid data meta-alert<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly=""> rules: - alert: ExampleAlert expr: vector_x + vector_y - alert: ExampleAlertInvalidData expr: absent(count(vector_x) == count(vector_x + vector_y))</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code> rules: - alert: ExampleAlert expr: vector_x + vector_y - alert: ExampleAlertInvalidData expr: absent(count(vector_x) == count(vector_x + vector_y))</code></pre></div> These are just a few examples of meta-alerts in HeraclesQL. Both contexts and annotations are just Python code that processes the expression’s AST. It’s possible to write custom implementations that can do just about anything you can imagine! <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop" id="h-pulling-it-together-jenkins-alerts">Pulling it Together – Jenkins Alerts</h2> HeraclesQL’s config library brings all of these features together to make a powerful code-as-configuration system. As a more realistic example, let’s write a small library for alerting based on the outcome of Jenkins jobs using the Jenkins Prometheus plugin: <a href="https://plugins.jenkins.io/prometheus/">https://plugins.jenkins.io/prometheus/</a> We’ll work under the assumption that we have multiple teams that want to alert on the status of their jobs. Jobs can do many things — they can be periodic jobs, they can be builds, and they can be manually triggered one-offs. Using plain VictoriaMetrics config, we’re likely to end up with a lot of near-duplicate alerts. A basic alert might look like this: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Example jenkins alert<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">rules: - alert: JenkinsJobFailing expr: default_jenkins_builds_last_build_result{job_name="Org/Team/Job"} == 0 for: 10m annotations: summary: Job {$labels.job_name} is failing for the last 10m</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>rules: - alert: JenkinsJobFailing expr: default_jenkins_builds_last_build_result{job_name="Org/Team/Job"} == 0 for: 10m annotations: summary: Job {$labels.job_name} is failing for the last 10m</code></pre></div> However, there’s already a problem — Jenkins registers metrics dynamically. If the job doesn’t exist for some reason (like bad configuration or a branch in a repo being deleted), this alert will not fire. So ideally we’ll check for absence as well: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Improved jenkins alert <pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">rules: - alert: JenkinsJobFailing expr: |- (default_jenkins_builds_last_build_result{job_name="Org/Team/Job"} == 0) or absent(default_jenkins_builds_last_build_result{job_name="Org/Team/Job"}) for: 10m annotations: summary: Job {$labels.job_name} is failing for the last 10m</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>rules: - alert: JenkinsJobFailing expr: |- (default_jenkins_builds_last_build_result{job_name="Org/Team/Job"} == 0) or absent(default_jenkins_builds_last_build_result{job_name="Org/Team/Job"}) for: 10m annotations: summary: Job {$labels.job_name} is failing for the last 10m</code></pre></div> Hopefully everyone always remembers to add that! There’s also one more complication: The Jenkins Prometheus plugin exports two metrics describing the last build result — <code>default_jenkins_builds_last_build_result</code> and <code>default_jenkins_builds_last_build_result_ordinal</code>. Both metrics are gauges that represent the status as a numerical value. The non-ordinal version uses 0 for a set of failed states and 1 for success. The ordinal version represents each state as a separate value (0-4). Confusingly, success is 0!  With many teams writing alerts for Jenkins jobs, this causes a few annoyances: <ol class="wp-block-list hrtbeatbulletspacer"> <li class="hrtbeatbulletspacer">Some alerts will use <code>default_jenkins_builds_last_build_result</code> and some will use <code>default_jenkins_builds_last_build_result_ordinal</code> even if they semantically find the same condition</li> <li class="hrtbeatbulletspacer">It’s easy to mistakenly check for the wrong values from <code>default_jenkins_builds_last_build_result_ordinal</code> since they’re different from the result from <code>default_jenkins_builds_last_build</code> and they’re just numeric values in code.</li> </ol> We found ourselves in this situation at HRT. Some alerts used <code>_ordinal</code>, some alerts forgot the absent check, and some alerts checked for the wrong result codes.  If you’ve read the previous section, you may already have some ideas about how Heracles can improve this. At the very least, we can use <code>AlertForMissingData</code> to ensure that nobody forgets the absent check anymore: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Jenkins alert in HeraclesQL<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() with rules.context(config.AlertForMissingData()): @rules.alert() def jenkins_job_failing() -> config.Alert: return config.SimpleAlert( expr=(v.default_jenkins_builds_last_result(jenkins_job="Org/Team/Job") < 1), for_=15 * ql.Minute, annotations={ "summary": "Job is failing for 15m", }, )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() with rules.context(config.AlertForMissingData()): @rules.alert() def jenkins_job_failing() -> config.Alert: return config.SimpleAlert( expr=(v.default_jenkins_builds_last_result(jenkins_job="Org/Team/Job") < 1), for_=15 * ql.Minute, annotations={ "summary": "Job is failing for 15m", }, )</code></pre></div> Next, we can add parameters to this alerting function so that we can define multiple alerts by repeating the <code>@rules.alert</code> annotations: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Alerts for multiple jobs<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.selector() with rules.context(config.AlertForMissingData()): @rules.alert("JobFrequent", 15 * ql.Minute) @rules.alert("JobInfrequent", 12 * ql.Hour) @rules.alert("JobImportant", 20 * ql.Minute) def jenkins_job_failing(job: str, duration: ql.Duration) -> config.Alert: return config.SimpleAlert( expr=v.default_jenkins_builds_last_result_ordinal( jenkins_job=job_name=f"Org/Team/{job}" ) > 1, for_=duration, annotations={ "summary": f"{job} is failing for {duration}", } )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.selector() with rules.context(config.AlertForMissingData()): @rules.alert("JobFrequent", 15 * ql.Minute) @rules.alert("JobInfrequent", 12 * ql.Hour) @rules.alert("JobImportant", 20 * ql.Minute) def jenkins_job_failing(job: str, duration: ql.Duration) -> config.Alert: return config.SimpleAlert( expr=v.default_jenkins_builds_last_result_ordinal( jenkins_job=job_name=f"Org/Team/{job}" ) > 1, for_=duration, annotations={ "summary": f"{job} is failing for {duration}", } )</code></pre></div> There’s still one major limitation with this implementation. As written, we will raise an alert for every job when it’s in <code>FAILURE</code> or <code>UNSTABLE</code> status. While that’s probably fine for most alerts, it might not be for some import alerts. For example, perhaps <code>JobImportant’s</code> alert is supposed to fire even when the job’s ordinal result is 1 (for <code>UNSTABLE</code>). Since we can’t use the same threshold for every alert, we need to add a parameter for build status and generate a more complex expression: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Alerts for multiple jobs with different statuses<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() with rules.context(config.AlertForMissingData()): @rules.alert("JobFrequent", duration=15 * ql.Minute) @rules.alert("JobInfrequent", duration=12 * ql.Hour) @rules.alert("JobImportant", duration=20 * ql.Minute, failure_status=(1, 2, 3, 4)) def jenkins_job_failing( job: str, duration: ql.Duration, failure_status: Iterable[int] = (2, 3), ) -> config.Alert: selector = v.default_builds_last_result_ordinal(jenkins_job=f"Org/Team/{job}") sub_exprs = [selector == s for s in failure_status] final_expr = functools.reduce(ql.InstantVector.or_, sub_exprs) return config.SimpleAlert( expr=final_expr, for_=duration, annotations={ "summary": f"{job} is failing for 15m", }, )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() with rules.context(config.AlertForMissingData()): @rules.alert("JobFrequent", duration=15 * ql.Minute) @rules.alert("JobInfrequent", duration=12 * ql.Hour) @rules.alert("JobImportant", duration=20 * ql.Minute, failure_status=(1, 2, 3, 4)) def jenkins_job_failing( job: str, duration: ql.Duration, failure_status: Iterable[int] = (2, 3), ) -> config.Alert: selector = v.default_builds_last_result_ordinal(jenkins_job=f"Org/Team/{job}") sub_exprs = [selector == s for s in failure_status] final_expr = functools.reduce(ql.InstantVector.or_, sub_exprs) return config.SimpleAlert( expr=final_expr, for_=duration, annotations={ "summary": f"{job} is failing for 15m", }, )</code></pre></div> Great! Now the expression is written in terms of a set of explicit statuses we want to alert on. By overriding the value of <code>failure_status</code> for <code>JobImportant</code>, we are able to alert when it is in anything other than a success status. However, now our alert definitions depend on passing around opaque integer statuses. That’s a step backwards in readability and makes it more likely that new alerts will use the wrong statuses by mistake. Luckily, HeraclesQL config is Python code! We can just add a new type: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Modelling Jenkins statuses in Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">import enum class JenkinsJobResult(int, enum.Enum): success = 0 unstable = 1 failure = 2 not_built = 3 aborted = 4 @staticmethod def success_statuses() -> Iterable["JenkinsJobResult"]: return (JenkinsJobResult.success, JenkinsJobResult.unstable) @staticmethod def failure_statuses() -> Iterable["JenkinsJobResult"]: return ( JenkinsJobResult.failure, JenkinsJobResult.not_built, JenkinsJobResult.aborted, )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>import enum class JenkinsJobResult(int, enum.Enum): success = 0 unstable = 1 failure = 2 not_built = 3 aborted = 4 @staticmethod def success_statuses() -> Iterable["JenkinsJobResult"]: return (JenkinsJobResult.success, JenkinsJobResult.unstable) @staticmethod def failure_statuses() -> Iterable["JenkinsJobResult"]: return ( JenkinsJobResult.failure, JenkinsJobResult.not_built, JenkinsJobResult.aborted, )</code></pre></div> And we can update the alert to use the new enum: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Using the new Jenkins status model<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">import functools from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() with rules.context(config.AlertForMissingData()): @rules.alert("JobFrequent", duration=15 * ql.Minute) @rules.alert("JobInfrequent", duration=12 * ql.Hour) @rules.alert( "JobImportant", duration=20 * ql.Minute, failure_status=( JenkinsJobResult.unstable, JenkinsJobResult.failure, JenkinsJobResult.not_built, JenkinsJobResult.aborted, ), ) def jenkins_job_failing( job: str, duration: ql.Duration, failure_status: Iterable[ JenkinsJobResult ] = JenkinsJobResult.failure_statuses(), ) -> config.Alert: selector = v.default_builds_last_result_ordinal(jenkins_job=f"Org/Team/{job}") sub_exprs = [selector == s.value for s in failure_status] final_expr = functools.reduce(ql.InstantVector.or_, sub_exprs) return config.SimpleAlert( expr=final_expr, for_=duration, annotations={ "summary": f"{job} is failing for 15m", }, )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>import functools from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() with rules.context(config.AlertForMissingData()): @rules.alert("JobFrequent", duration=15 * ql.Minute) @rules.alert("JobInfrequent", duration=12 * ql.Hour) @rules.alert( "JobImportant", duration=20 * ql.Minute, failure_status=( JenkinsJobResult.unstable, JenkinsJobResult.failure, JenkinsJobResult.not_built, JenkinsJobResult.aborted, ), ) def jenkins_job_failing( job: str, duration: ql.Duration, failure_status: Iterable[ JenkinsJobResult ] = JenkinsJobResult.failure_statuses(), ) -> config.Alert: selector = v.default_builds_last_result_ordinal(jenkins_job=f"Org/Team/{job}") sub_exprs = [selector == s.value for s in failure_status] final_expr = functools.reduce(ql.InstantVector.or_, sub_exprs) return config.SimpleAlert( expr=final_expr, for_=duration, annotations={ "summary": f"{job} is failing for 15m", }, )</code></pre></div> Much better! Now it’s impossible for users to forget the mapping between logical status and numeric code. What if we want to share this alert between multiple teams? We can just add more decorators to <code>jenkins_job_failing</code>, but that’ll get hard to read pretty fast. Luckily, I have one more trick up my sleeve: HeraclesQL has an alert extension syntax: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Extending a rule<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">rules.alert( "JobFrequentFailing", rules.extends(jenkins_job_failing, duration=15 * ql.Minute) )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>rules.alert( "JobFrequentFailing", rules.extends(jenkins_job_failing, duration=15 * ql.Minute) )</code></pre></div> Now we can update our rules so that new alerts are more declarative: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Declarative alert generation<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() jenkins_alerts = { "JobFrequent": {"duration": 15 * ql.Minute}, "JobInfrequent": {"duration": 12 * ql.Hour}, "JobImportant": { "duration": 20 * ql.Minute, "failure_status": ( JenkinsJobResult.unstable, JenkinsJobResult.failure, JenkinsJobResult.not_built, JenkinsJobResult.aborted, ), }, } with rules.context(config.AlertForMissingData()): for job, params in jenkins_alerts.items(): rules.alert( f"{job}Failing", rules.extends(jenkins_job_failing, **params), )</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from heracles import ql, config rules = config.RuleBundle("hrtbeat.rules.yml") v = rules.vectors() jenkins_alerts = { "JobFrequent": {"duration": 15 * ql.Minute}, "JobInfrequent": {"duration": 12 * ql.Hour}, "JobImportant": { "duration": 20 * ql.Minute, "failure_status": ( JenkinsJobResult.unstable, JenkinsJobResult.failure, JenkinsJobResult.not_built, JenkinsJobResult.aborted, ), }, } with rules.context(config.AlertForMissingData()): for job, params in jenkins_alerts.items(): rules.alert( f"{job}Failing", rules.extends(jenkins_job_failing, **params), )</code></pre></div> And that’s it! New teams can even create new modules and extend this alert from there. This is much easier to extend than a VictoriaMetrics rule config file with a growing list of nearly duplicated rules. It’s also much more difficult to make mistakes — we’re able to use Python’s type system to avoid magic numbers and use HeraclesQL’s config APIs to automatically add meta-alerts to catch common mistakes. <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop">Development and Testing</h2> Our goal with HeraclesQL is to make the experience of developing alerts better. This extends beyond just the code you write to the experience of writing and testing code. To that end, I’d like to highlight two development tools we’re working on: Delos and Hermes. <h3 class="wp-block-heading hrtbeatentrytop">Delos</h3> Delos is a small development environment for writing HeraclesQL queries. When writing MetricsQL, it’s normal to iterate on queries inside the VictoriaMetrics UI or a similar tool which can graph query results on the fly. Developers lose this ability when writing HeraclesQL because those UIs don’t support HeraclesQL. Delos fills this gap by proxying a live instance of VictoriaMetrics with a query populated from a HeraclesQL Python project. <figure class="wp-block-image size-full hrtbeatentrytopandbot"><img loading="lazy" decoding="async" width="4096" height="2304" src="https://www.hudsonrivertrading.com/wp-content/uploads/2025/10/delos-demo.gif" alt="" class="wp-image-396668" referrerpolicy="no-referrer"></figure> Delos is still somewhat experimental – it works, but there are plenty of rough edges. For this reason, it’s not included in the wheel just yet (but it can be installed directly as a Go module). Delos has a very simple implementation. The UI is provided by an iframe that embeds the VictoriaMetrics UI from a target VictoriaMetrics instance. Your HeraclesQL query is injected into the UI via a small HTTP API in the Delos server, which executes a Python script that generates MetricsQL from your query. Using Delos is easy: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Delos interactive mode<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">$ delos --vm-url <a-vm-instance> Found virtual env at '/home/you/.venv' Found python at '/home/you/.venv/bin/python' -- Delos Interactive Mode -- -> http://localhost:5000/dev <- QUERY: 42.0 Enter command: 'q' - exit 'e' - edit</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ delos --vm-url <a-vm-instance> Found virtual env at '/home/you/.venv' Found python at '/home/you/.venv/bin/python' -- Delos Interactive Mode -- -> http://localhost:5000/dev <- QUERY: 42.0 Enter command: 'q' - exit 'e' - edit</code></pre></div> Press ‘e’ to open the Delos query file. If you’re using a <a href="https://docs.python.org/3/library/venv.html">Python virtual env</a>, this file will be embedded in that environment so you can reference modules from your project.  <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Delos query file<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly=""># Use this file to define the query which Delos will display! # This script executes inside the provided venv, so code from your project is directly callable. # # Delos expects a variable named 'VECTOR' which contains a query. `VECTOR` can also be a function. from heracles import ql VECTOR = ql.ScalarLiteral(42)</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code># Use this file to define the query which Delos will display! # This script executes inside the provided venv, so code from your project is directly callable. # # Delos expects a variable named 'VECTOR' which contains a query. `VECTOR` can also be a function. from heracles import ql VECTOR = ql.ScalarLiteral(42)</code></pre></div> So for example, to view an alert query, do this: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Referencing project modules from Delos<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly=""># Use this file to define the query which Delos will display! # This script executes inside the provided venv, so code from your project is directly callable. # # Delos expects a variable named 'VECTOR' which contains a query. `VECTOR` can also be a function. from heracles import ql from my_heracles_rules import my_team_rules VECTOR = my_team_rules.MyUsefulAlert.expr</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code># Use this file to define the query which Delos will display! # This script executes inside the provided venv, so code from your project is directly callable. # # Delos expects a variable named 'VECTOR' which contains a query. `VECTOR` can also be a function. from heracles import ql from my_heracles_rules import my_team_rules VECTOR = my_team_rules.MyUsefulAlert.expr</code></pre></div> Save this file, and Delos will automatically populate the query into the UI. When you edit MyUsefulAlert, Delos will automatically update the query in the UI. We think Delos provides a better experience than manually writing a rules.yaml file because you can iterate on alerts directly from the definitions. With a rules.yaml, you still have to copy alerts between the VictoriaMetrics UI and the rules.yaml file.  <h3 class="wp-block-heading hrtbeatentrytop">Hermes</h3> If you’re familiar with writing PromQL or MetricsQL alerts, you’re probably familiar with promtool test and vmalert-tool test. These commands allow rules to be tested against a real timeseries database with synthetic data. It’s still possible to use these tools with HeraclesQL generated alerts, but we wanted to provide a better experience.   Hermes re-implements the behavior of vmalert-tool with a programmatic interface. Python code can generate synthetic timeseries, write them to the Hermes client, execute a query, and read back the results. This can be used to implement unit tests for HeraclesQL queries that actually test against a real VictoriaMetrics server. Unlike vmalert-tool tests, Hermes tests can use imperative code to generate input timeseries. Similarly, Hermes returns the raw query result to the client, allowing imperative logic for checking conditions of the result. This makes it much easier to test complex conditions. Hermes is still early in its development, so it has some rough edges. However, we think it’ll be a great mechanism for fully native HeraclesQL testing as it improves. <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop" id="h-wrapping-up">Wrapping Up</h2> We’re very excited about the potential of HeraclesQL. We’re already using it for many alerts inside HRT, and we plan to eventually replace all of our MetricsQL alerts with it. HeraclesQL isn’t done yet, either — we plan to continue developing the project, especially now that it’s available to the community as an <a href="https://github.com/hudson-trading/heracles-ql#">open source project!</a> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"> <h2 class="wp-block-heading hrtbeatentrytop">Future Possibilities</h2> <h3 class="wp-block-heading hrtbeatentrytop" id="h-prometheus-support">Prometheus Support</h3> HeraclesQL supports MetricsQL because we only use VictoriaMetrics at HRT. However, it’d be entirely possible to support Prometheus’ PromQL if there’s community interest! HeraclesQL was written so that the core types would be easy to swap.  <h3 class="wp-block-heading hrtbeatentrytop">Execution Support</h3> HeraclesQL supports generating query strings and generating static configs. This is what’s necessary for alerting, but VictoriaMetrics can be used in many other cases. In the future, we could add support for executing HeraclesQL queries in the same way one can execute a SQLAlchemy query.

2025/10/28

Answers to Questions I Often Get: Engineering and Interviewing at HRT

<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"> <h3 class="wp-block-heading" id="h-it-s-been-over-a-decade-since-i-last-wrote-something-about-hrt-a-lot-has-changed-since-then-hrt-is-much-larger-and-we-trade-in-many-more-venues-and-in-many-different-time-horizons-but-i-still-feel-like-the-fundamental-vibe-is-thankfully-similar" style="font-style:normal;font-weight:400">It’s been over a decade since <a href="https://www.quora.com/What-is-it-like-to-work-at-Hudson-River-Trading" target="_blank" rel="noreferrer noopener">I last wrote something about HRT</a>. A lot has changed since then. HRT is much larger, and we trade in many more venues and in many different time horizons. But I still feel like the fundamental vibe is, thankfully, similar.</h3> </blockquote> We do now have a much wider variety of engineering jobs, which, generally speaking, is a good thing. But it can be a little difficult for someone interested in engineering at HRT to figure out what exactly is happening internally and where they may best fit. I’ve found myself repeatedly giving the same “spiel” during interviews, and I figured it’d make sense to just write it down so everyone has access to the same information. The remainder of this post should help you get a better mental model of the topology of engineering at HRT and then explain how to approach the interview. Please keep in mind that I cannot possibly address every specialty at HRT in a single blog post. So if you have a skill set that would work at HRT and it’s not explicitly mentioned here, please still apply, and our recruiting team will help you find the right fit. And finally, this blog post is only about engineering. Algo Development (aka quantitative research and trading), People Operations, Accounting, and numerous other job families are critical to HRT’s daily operations, but in this post I’m going to stick to what I know best. <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-engineering-roles">Engineering Roles</h2> Before I dive into engineering role topology, I want to briefly describe how Algo (trading teams) work, because ultimately they are the customer all engineers work to support. Algo is composed of a bunch of different trading teams. They are very roughly organized by the time horizon at which they trade, and then by asset class and region. People in Algo mostly work on researching various models and algorithms used to trade on electronic markets. They also actively deploy and monitor their trading. One very important thing to note about Algo: it is not a “pod” system where trading teams are siloed with IP barriers. This is especially important for engineers because it means you can work across all trading at the firm rather than being contained to some specific group. If you come from big tech this may not seem all that interesting, but if you come from finance, you’ll know that this is huge for engineering.  At a very high level, our engineering is divided into two major groups: Trading Tech and Research & Development. Trading Tech is primarily concerned with what needs to be done in order to actually trade, with special focus on “today”. This includes parsing high speed data from exchanges, placing orders with exchanges, applying hardware acceleration (FPGA/ASIC), processing trades, clearing, reconciliation, etc. Research & Development is primarily concerned with enabling research with focus on “history” (versus Trading Tech’s “today”). This includes storage, data centers, CPU/GPU clusters, job scheduling/orchestration, data ETL, and research tools for the Algo group (our trading teams). I’d say that Trading Tech’s language focus is something like 70% C++ and 30% python, and that Research & Development’s is roughly 70% python and 30% C++. The quality bar is identical: R&D’s C++ teams are as serious about C++ as Trading Tech’s, and Trading Tech’s Python teams are as serious about Python as R&D’s. Trading Tech’s biggest technical challenge is “latency,” whereas Research & Development’s biggest technical challenge is “scale”. Each of these two divisions has both Acceleration and Platform teams. The engineering group dedicated to working most directly with specific Algo teams is formally called “Acceleration” (e.g. Trading Acceleration and Research Acceleration), because their job is to accelerate our trading teams’ activities. Informally you will often hear this type of job  referred to as “an embedded engineering” job. The idea is that the engineer is “embedded into” the trading team, often aligned with a particular trading strategy or specific sub-team of Algo Developers. We refer to everything that is not “Acceleration” as “Platform.” Broadly speaking, platform engineers are dedicated to one or more products that are used across all of Algo. Typically, platform teams work on a longer time horizon and plan product roadmaps that take many different trading teams’ requirements into account, whereas the Acceleration teams work on more iterative tasks and help customize Platform products for their specific trading team’s needs. One thing to note that may be different from other companies, especially other trading firms: our platform engineers interact with Algo on a daily basis. The intention is not to put Platform behind a layer, it’s more about project focus.  <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="577" height="387" src="https://www.hudsonrivertrading.com/wp-content/uploads/2025/10/image.png" alt="" class="wp-image-396511" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2025/10/image.png 577w, https://www.hudsonrivertrading.com/wp-content/uploads/2025/10/image-480x322.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 577px, 100vw" referrerpolicy="no-referrer"><figcaption class="wp-element-caption"> This is a picture of what I’ve described so far (engineering roles in gray)</figcaption></figure> So far I’ve discussed two dimensions we used to describe engineering roles: research vs live trading and platform vs acceleration. The final dimension I use to describe engineering at HRT is programming language skills. I actually was going to describe this as systems vs application engineering, or low-level vs high level programming, but I think the language requirements for a specific role are a good proxy for this and are less abstract. Generally speaking, if you interview in C++ at HRT, you are expected to have a fair bit of low-level knowledge (i.e. how computers work). If you interview in Python, you are expected to still have some systems knowledge (e.g. understand how virtual memory works) but less specifics than the C++ jobs. And some jobs are a mix. There are also some jobs where the systems knowledge is especially important, like hardware/software interfaces, kernel programming, or systems automation skills. So I’m just going to count that as its own language skill (Low Level C++) even though it’s not a perfect abstraction. Broadly speaking, a given job can be one of the following, in rough order of low level to high level: <ul class="wp-block-list"> <li class="hrtbeatbulletspacer">Verilog (Hardware/FPGA)</li> <li class="hrtbeatbulletspacer">Low Level C++ (LLC++)</li> <li class="hrtbeatbulletspacer">C++</li> <li class="hrtbeatbulletspacer">C++/Python</li> <li class="hrtbeatbulletspacer">Python</li> <li class="hrtbeatbulletspacer">Systems Automation/Python</li> <li class="hrtbeatbulletspacer">Typescript/Python</li> </ul> At this point I want to note something very important (to me anyway): these dimensions describe jobs at HRT, but many people are capable of performing multiple jobs over their career and we actively encourage people to “rotate” between different jobs, across all of these dimensions. My favorite example is that I had someone start in a Research Accel job many years ago <kbd>[Research | Accel | Python]</kbd> and he expressed interest in FPGA programming. He is now the group lead for the hardware/software integration team <kbd>[Live Trading | Platform | LLC++]</kbd>. He was great at both jobs. That being said, if you are applying to HRT, I encourage you to start in a job where you will have the fastest immediate impact. That is—start with your strength. The more impact you have, the easier it is to move around.  <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-specific-examples">Specific Examples</h2> Now I’d like to take a few examples (not exhaustive!) of various engineering roles across HRT and map them to these dimensions: <ul class="wp-block-list"> <li class="hrtbeatbulletspacer">Distributed Compute Engineer. This is a job working on the scheduling and orchestration software that runs our clusters. It’s probably about 60% C++ and 40% Python. This falls within the R&D division, but is a platform job (since all Algo teams use this software) and it’s a mix of systems and application programming. <kbd>[Research | Platform | C++/Python] </kbd></li> <li class="hrtbeatbulletspacer">HFT Research Accel Engineer. An engineering role to help with the research and post trade analysis for our high frequency trading group. <kbd>[Research | Accel | Python]</kbd></li> <li class="hrtbeatbulletspacer">BAT Engineer (Build Autotest Tools). This engineer would work on our internal development tools platforms, which include large scale distributed test systems (aka “autotest”), distributed and cached builds, and many more tools used by all engineering and trading teams at HRT. <kbd>[Research | Platform | C++/Python] </kbd></li> <li class="hrtbeatbulletspacer">Fullstack Engineer. Our fullstack team happens to live within Trading Tech since we have more custom live trading GUIs than we do custom research GUIs. Arguably this team could span both Trading Tech and R&D – maybe it will one day! For now this role requires typescript and python <kbd>[Live Trading | Platform | Typescript/Python]</kbd></li> <li class="hrtbeatbulletspacer">Core Markets Engineer. An engineer here will work on order entry and market data connectivity for particular exchanges across a variety of asset classes, in addition to improving the underlying platform and frameworks that back it <kbd>[Live Trading | Platform | C++] </kbd></li> <li class="hrtbeatbulletspacer">Trading Tech Infrastructure Engineer. Engineers on this team might work on: the infrastructure responsible for routing data between datacenters across low-bandwidth, low-latency links, or on the software that interfaces with the hardware used for order entry and market data  <kbd>[Live Trading | Platform | Low Level C++] </kbd></li> <li class="hrtbeatbulletspacer">Options Trading Accel Engineer. Devs on this team support our options market making trading team; they implement market data + order entry protocols to global option exchanges, solve latency & scaling issues in the live trading path, and also work on strategy & execution logic <kbd>[Live Trading | Accel | C++]</kbd></li> <li class="hrtbeatbulletspacer">Systems Software Engineer. This team maintains the software platform that is used to provision, manage, and monitor HRT’s servers and network equipment. This platform is shared between our live trading and research environments, so roles on this team span both worlds. The work is primarily Python development (with some Go mixed in), but also deploying and running high-reliability services. <kbd>[Live trading and Research | Platform | Systems Automation/Python]</kbd></li> </ul> If you navigate to <a href="https://www.hudsonrivertrading.com/careers/">our careers page</a>, what you’ll see is a number of open roles on the website for a number of different teams, some of which were mentioned earlier in this post.  We recommend that you apply to the one that most closely mirrors your expertise and your interests. That said, we have most roles flowing through the same pipeline so we can adjust where you’re headed once you’re further along in the interview process. <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-what-is-an-interview-like">What is an interview like?</h2> The interview process begins with a series of Zoom calls. Depending on the role and the candidate, these can be exploratory conversations or jump right into technical phone screens. Eventually all candidates will go through technical interviews.  Right now you can choose between C++, Python, or Typescript/Python (for Fullstack) as your programming language of choice. I should note that you may be given problems that require you to analyze code in an unfamiliar language to gauge your ability to pattern-match and adapt (but when we grade these problems, we take candidates’ language familiarity into consideration).  If things continue to go well, you will be invited to onsite interviews.  We much prefer at least some of the onsite interviews to be in person (which is also good for you to get a better feel for HRT). The interviews vary slightly between some roles, but they are generally composed of coding and debugging rounds, technical design discussions, and team fit.  We do our best to stray away from “burst of insight” leet-code style questions. Our ideal would be that a strong programmer from a competitive firm would ace our technical interview with no studying. We want engineers with strong fundamentals but stay away from obscura and “tricks”. Of course what constitutes obscura is subjective, but you can at least have an idea what our intentions are. I will say that most people do not fail the interview because they lack some specialized piece of knowledge. That brings me to the next section. <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-common-interview-pitfalls">Common Interview Pitfalls</h2> I’ve been asked hundreds of times for tips on interviewing, so I’m going to be as transparent as I possibly can. Instead of listing a bunch of generic sounding traits, let’s just assume we want the best engineers in the world, and I’ll list the most common reasons I see candidates fail interviews (in rough order of frequency): <ul class="wp-block-list"> <li class="hrtbeatbulletspacer">A shallow understanding of fundamentals. Many people use tools, libraries, techniques, and math and don’t have a true intuition for how they work. Often, the true nature of these tools is rooted in how they originated: why they were invented or designed in the first place. It’s not about esoteric knowledge, it’s about truly understanding something inside and out. If you use virtual classes, do you know how most compilers implement them? If you use python lists, do you know the memory growth characteristic as it expands? </li> <li class="hrtbeatbulletspacer">Communication issues. If you think that software engineering isn’t just as much about communicating to your peers and your users as it is about writing code, then HRT is not the right place for you. We rate our engineers as much on communication as we do on technical ability. This is the ability not only to clearly articulate your thoughts but also to understand what others are thinking, which leads us to the next pitfall. </li> <li class="hrtbeatbulletspacer">Listening and Understanding. I see so much interview feedback that says something like “the candidate didn’t fully understand the problem before they dove in” or “I couldn’t get a word in edge-wise”. This is not only a technical red flag, but also can be an early warning sign that someone isn’t accustomed to practicing empathy and understanding the people around them. It doesn’t matter how good your technical skills are if you cannot understand the human context around the technical work.</li> <li class="hrtbeatbulletspacer">Avoiding Real Work. In an interview, this could appear as candidates avoiding tackling problems directly or proposing overly complex solutions instead of simply coding a practical answer. At HRT we value individual contribution. We expect engineers and engineering managers to make individual contributions on a quarterly basis. This is why we administer the same technical tests to candidates at all levels, even if you were the CTO of your last company. Anecdotally, the Architect of Research & Development was formerly a CTO, and he can code circles around most engineers. Expect to program during the interview and expect to do measurable technical work on the job, regardless of your level. The best engineers at HRT are thriving because they regard this as one of the biggest perks of the job as opposed to a requirement. </li> <li class="hrtbeatbulletspacer">Arrogance. I think this should speak for itself, but hey— this is finance. We get world-class candidates every day, and while we admire their achievements, we expect them to conduct themselves with a grace commensurate with their reputation. HRT is full of people that were the top of their class, their company, and their industry. The highest performers are those who make everyone around them better, not those who stand apart.</li> <li class="hrtbeatbulletspacer">Indifference. I often see feedback like “the candidate just doesn’t seem that excited”. We are not looking for people that are bubbly cheerleaders for HRT (hey – they already have me 🙂 ). But we want people that illustrate some passion about their work. This can be as subtle as pride in the quality of their work. Someone that is going to just treat this job like any other job should probably just get any other job.</li> <li class="hrtbeatbulletspacer">Cheating. Some candidates attempt to cheat the earlier stages by using LLMs and the like, only to find that the later stages are done in a manner that makes cheating significantly more difficult, and then everyone’s time is wasted. We do not welcome cheating candidates to interview again at a later time.</li> </ul> <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-conclusion">Conclusion</h2> I intend for this post to be for people who are already interested in HRT and want to understand more about our roles and the interview process and so I didn’t include a whole lot of “sell” in this post. Bear with me for just a teeny bit of cheerleading. At the time of this writing, I’ve been with HRT for about 13 years. I have worked in big tech, gaming, government, and finance. This is, by a wide margin, the best company I have worked for. It is not perfect, but it is great, and it is great because of the people who make up HRT. Our interview process heavily screens for people who are not only brilliant, but also open, curious, kind, practical, and actually enjoy working. I’m not entirely sure why you’d have read all the way to this point if you weren’t already sold on HRT, but if you have and this sounds like you, please do apply! -Joe

2025/10/2

Building a Distributed Filesystem for Scalable Research

<blockquote class="wp-block-quote is-style-default hrtbeatentrytop is-layout-flow wp-block-quote-is-layout-flow"> <h3 class="wp-block-heading" id="h-the-ability-to-store-and-analyze-vast-amounts-of-data-is-table-stakes-for-an-algorithmic-trading-firm-like-hrt-some-of-our-storage-requirements-weren-t-well-met-by-existing-solutions-so-we-decided-to-build-and-operate-our-own-distributed-filesystem-called-blobby" style="font-style:normal;font-weight:400">The ability to store and analyze vast amounts of data is table stakes for an algorithmic trading firm like HRT. Some of our storage requirements weren’t well met by existing solutions, so we decided to build and operate our own distributed filesystem, called Blobby.</h3> </blockquote> <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" id="h-why" style="font-style:normal;font-weight:600">Why?</h2> HRT ingests a large amount of data. <a href="https://www.hudsonrivertrading.com/hrtbeat/how-hrt-thinks-about-data/">Algo researchers</a> then use the data to develop and test their ideas, and these ideas can turn into trading strategies. Large-scale data is core to our business. Much of HRT’s infrastructure resides on-premises. We have historically relied on a mix of storage solutions, such as <a href="https://www.lustre.org/">Lustre</a>, <a href="https://en.wikipedia.org/wiki/Network_File_System">NFS</a> filers, and various open-source engineered in house options. These have supported our workloads in the past, but as HRT’s storage needs have grown, we’ve hit significant limitations: <ul class="wp-block-list"> <li>Cost. Purchasing storage appliances from third-party vendors is expensive when storing hundreds of petabytes of data.</li> <li>Scale. At the scale we are building, HRT would be one of the largest users of open-source storage systems such as <a href="https://ceph.com/community/new-luminous-scalability/">Ceph</a> or <a href="https://community.hpe.com/t5/servers-systems-the-right/meet-the-world-s-largest-and-fastest-parallel-file-system/ba-p/7155645">Lustre</a>.</li> <li>Scalability. Scaling our existing storage solutions has often been cumbersome, and in some cases, outright impossible. We wanted a system that can scale with minimal operational or user burden simply by adding commodity hardware.</li> <li>Multi-tenancy. HRT is home to many different teams and researchers with different workflows. These users need to be able to run experiments and train models without being disrupted by each other. Historically, we’ve dealt with noisy neighbors by limiting the concurrency of their jobs and in extreme cases allocating them their own isolated storage system. While partitioning resources is effective at isolation, it has led to stranded storage capacity and bandwidth, racking up considerable extra costs and operational overhead.</li> <li>Control. Owning the entire storage stack gives us full control to prioritize the properties that matter most to HRT. We can design the system to specifically meet the needs of our workloads, as well as fix issues in a timely manner instead of going through lengthy third-party debugging/fixing loops.</li> </ul> To overcome these limitations, we decided to build Blobby, our distributed filesystem. We are not the only ones to have taken this path: DeepSeek has <a href="https://github.com/deepseek-ai/3FS">3FS</a>, Meta has <a href="https://www.usenix.org/system/files/fast21-pan.pdf">Tectonic</a>, Google has <a href="https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system">Colossus</a>.  <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" style="font-style:normal;font-weight:600">Design Principles and Requirements</h2> Building and operating a storage system is a significant undertaking. Large tech companies employ many engineers to build and operate such a system. HRT is a tiny firm compared to the likes of Google, Meta, or Amazon. Designing and developing Blobby with a very small team has required us to lean heavily on prior art and focus on the minimal set of requirements for HRT workloads: <ul class="wp-block-list"> <li>High aggregate throughput. Blobby is not designed for maximum single-client performance, but for sustained, high aggregate throughput across many concurrent clients. </li> <li>Durability. At scale, hardware failure is common, but data loss is unacceptable.  Disks fail, network links flap, software upgrades go wrong – user data has to be safeguarded against these inevitable failures.  We run continual “chaos testing” to ensure that Blobby safeguards data under complex failure scenarios.</li> <li>Simplicity and maintainability. As a small team, we have been deliberate about introducing complexity only when required. For example, because most HRT workloads write large files in a single pass, Blobby is append-only.  If future clients need in-place updates, we would build that on top of Blobby as another layer.</li> </ul> <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" style="font-style:normal;font-weight:600">Architecture</h2> Much of our system is based on prior art like <a href="https://www.usenix.org/system/files/fast21-pan.pdf">Meta’s Tectonic</a> filesystem. A high-level picture: <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="537" height="381" src="https://www.hudsonrivertrading.com/wp-content/uploads/2025/08/image1.png" alt="" class="wp-image-396191" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2025/08/image1.png 537w, https://www.hudsonrivertrading.com/wp-content/uploads/2025/08/image1-480x341.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 537px, 100vw" referrerpolicy="no-referrer"><figcaption class="wp-element-caption"> Boxes represent individual processes running on different “machines”,  typically all within the same data center. Some of these processes run on bare-metal machines, some on virtual machines, and some in Kubernetes containers.  Arrows represent the direction in which communication is initiated, typically over TCP.</figcaption></figure> <h4 class="wp-block-heading">Data Layout</h4> Blobby files are divided into blocks of arbitrary size, and blocks are broken down into chunks. Blobby supports two schemes for splitting blocks into chunks: <ul class="wp-block-list"> <li><a href="https://en.wikipedia.org/wiki/Replication_(computing)">Traditional replication</a>, where the block’s data is written N times.</li> <li><a href="https://en.wikipedia.org/wiki/Erasure_code">Erasure coding</a>, where the block is broken up into N data chunks. Erasure coding is applied to generate K parity chunks. Any N chunks can be used to generate all other chunks. For a more detailed explanation, see <a href="https://www.backblaze.com/blog/reed-solomon/">Backblaze Open-sources Reed-Solomon Erasure Coding Source Code</a>.</li> </ul> Blobby’s default block layout is erasure coding with 6 data chunks and 3 code chunks; this allows a loss of any 3 chunks in a block without data loss. We also use a default chunk size of 8MB; each chunk read is a random read on a hard drive, and larger reads increase overall throughput. <h4 class="wp-block-heading hrtbeatentrytop">Clients</h4> The Blobby client is a C++ library that provides the following API: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">// Metadata list(path) -> [path] rm(path) rmdir(path) mkdir(path) // Reading open(path) -> reader pread(reader, off, len) -> bytes close(reader) // Writing create(path) -> writer append(writer, bytes) flush(writer) close(writer)</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>// Metadata list(path) -> [path] rm(path) rmdir(path) mkdir(path) // Reading open(path) -> reader pread(reader, off, len) -> bytes close(reader) // Writing create(path) -> writer append(writer, bytes) flush(writer) close(writer)</code></pre></div> This library is linked directly into end-user applications. Blobby distributes the chunks of a file across the entire cluster. The Blobby client library communicates with Blobby servers on behalf of the end-user application to turn client reads/writes into RPCs to the correct servers into reads/writes of the relevant chunks. Data only flows between client and chunkservers for linear scalability. <h4 class="wp-block-heading hrtbeatentrytop">Data Placement</h4> When we create a block, we have to choose where to store that block’s chunks. One approach is to randomly choose chunkservers. This results in uniform data distribution, but may suffer from an <a href="https://martin.kleppmann.com/2017/01/26/data-loss-in-large-clusters.html">increased probability of data loss at scale</a>. We instead rely on ideas from <a href="https://web.stanford.edu/~skatti/pubs/usenix13-copysets.pdf">Copysets: Reducing the Frequency of Data Loss in Cloud Storage</a>. Instead of choosing randomly for each chunk, we choose from a smaller set of groups, reducing the likelihood of data loss if many disks/servers fail at the same time.  We also ensure that each block’s chunks are in different racks to guard against data loss on rack failure. <h4 class="wp-block-heading hrtbeatentrytop">Chunk Servers</h4> Chunk servers store the actual chunk contents. Each chunk server has hundreds of terabytes of local storage attached. The API is intentionally simple: A chunkserver supports storing an entire chunk and reading a byte range from an existing chunk. Each chunk is represented as a file stored on XFS. While XFS has served us well as an initial implementation, there’s still room for improvement. As noted in <a href="https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf">Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution</a>, it may not be the best long-term choice for a distributed storage system. <h4 class="wp-block-heading hrtbeatentrytop">Metadata DB</h4> The metadata database stores: <ul class="wp-block-list"> <li>Filesystem metadata. Directory tree structure, the blocks which make up a file, the chunkservers on which chunks have been placed on, etc.</li> <li>Cluster metadata. Disk and chunk server health as well as service-specific configuration files.</li> </ul> We use <a href="https://www.foundationdb.org/">FoundationDB</a> for our metadata database. It has several powerful properties and features that make it useful: <ul class="wp-block-list"> <li><a href="https://apple.github.io/foundationdb/scalability.html">Horizontal scalability</a>: The metadata database runs on commodity machines and can serve more read/write traffic by adding machines. </li> <li><a href="https://apple.github.io/foundationdb/fault-tolerance.html">Fault tolerance</a>: Replication of the metadata database means Blobby remains available even if the metadata database suffers an individual server failure.  </li> <li><a href="https://apple.github.io/foundationdb/transaction-manifesto.html">ACID transactions</a>: This makes implementing operations which need to operate atomically over several rows (e.g. file deletion) trivial. As a result, we avoid needing a “garbage collector” for metadata inconsistencies like the one <a href="https://www.usenix.org/system/files/fast21-pan.pdf">Tectonic</a> describes.</li> <li><a href="https://apple.github.io/foundationdb/consistency.html">Strict serializability/consistency:</a> Like ACID transactions, this significantly decreases implementation complexity.</li> </ul> DeepSeek also, independently, decided to use FoundationDB for <a href="https://github.com/deepseek-ai/3FS/tree/main/src/fdb">3FS</a> metadata. <h4 class="wp-block-heading hrtbeatentrytop">Proxy Servers</h4> The proxy servers serve all non-data client operations. This includes: <ul class="wp-block-list"> <li>Metadata operations. For example, clients contact a proxy to learn which chunk server has the data for a particular range of file bytes. A proxy services these requests by reading/writing the filesystem metadata in the metadata database.</li> <li>Write steering. Clients contact a proxy to learn which disks to write chunks to. Proxies have a reasonably fresh view of cluster health, which allows them to steer writes towards writable disks. </li> </ul> Clients never talk directly to the metadata database. All reads/writes to the metadata database flow through a proxy server. This allows us to avoid buggy clients corrupting metadata, implement access controls, rate-limits or swap the metadata database under the hood when/if we need to. <h4 class="wp-block-heading hrtbeatentrytop">Background Services</h4> Our background services fulfill a variety of roles, from monitoring to failure detection and recovery. Clients do not directly interact with them, but they are vital to maintain the reliability of the system. <h5 class="wp-block-heading">Shepherd</h5> The shepherd maintains system health by: <ul class="wp-block-list"> <li>Detecting unhealthy servers and disks via regular health checks. Chunk servers constantly issue small reads and writes to their disks to determine disk health. The shepherd continuously sends status RPC requests to chunk servers, and considers a chunk server unhealthy if it does not reply for long enough.</li> <li>Updating cluster metadata to reflect the shepherd’s view of chunk server health. This allows: <ul class="wp-block-list"> <li>Proxies to steer new writes away from unhealthy servers/disks.</li> <li>The repair service to move existing data from unhealthy servers/disks to healthy ones.</li> </ul> </li> </ul> We currently run a single shepherd instance per cluster. While this technically introduces a single point of failure, temporary unavailability of the shepherd does not impact client operations, as it is not in the critical path for reads or writes. That said, if our operational needs evolve, we are not bound to this design choice. <h5 class="wp-block-heading">Repair</h5> The repair service is responsible for recovering chunks in response to hardware failure. It continuously scans metadata for partially lost blocks and prioritizes recovering blocks that can tolerate the fewest additional chunk losses.  To allow for horizontal scalability, each instance of the repair service operates on a separate shard of the block space. <h5 class="wp-block-heading">Prober</h5> Our training runs involve many jobs, each running on different hosts.  Each job synchronously loads data through parallel data loading processes on each host, trains on that data, and then shares results with other jobs.  A stall on any one data loading process will stall all jobs in the run, so it is very important to minimize tail latency. We initially relied on metrics generated by real clients to monitor this. We run a variety of workloads across noisy hosts, so client-side slowness cannot always be directly attributed to Blobby. In order to get trustworthy client-perceived latencies, we run probers, dedicated clients that continuously read and write to Blobby in a controlled environment.  <div class="wp-block-separator has-alpha-channel-opacity is-style-wide hrtbeatdiv"></div> <h2 class="wp-block-heading hrtbeatentrytop" style="font-style:normal;font-weight:600">Conclusion</h2> HRT currently has one Blobby cluster in “soft-production,” and is building out several additional clusters.  Our NVMe-based cluster is the highest-throughput durable storage system at HRT. Our HDD-based clusters offer very high throughput at the lowest cost-per-GB. We are actively migrating workloads to Blobby clusters. Blobby offers enough storage and performance that researchers simply don’t have to pay close attention to things like quota, and can instead focus entirely on their research. The storage-dev team is excited to see what research “nearly unlimited bytes” will enable. The storage-dev team is small but growing. If you are excited by foundational distributed systems, large-scale data pipelines, or low-latency compute infrastructure, we would love to hear from you. And if the name “Blobby” made you smile, you might be our kind of person. You can apply <a href="https://www.hudsonrivertrading.com/hrt-job/experienced-c-developer-distributed-systems-storage-databases/">here</a> or check out our <a href="https://www.hudsonrivertrading.com/careers">careers page</a> for all of our compute and storage roles. <h4 class="wp-block-heading">Sources</h4> <ul class="wp-block-list"> <li><a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf">Availability in Globally Distributed Storage Systems</a></li> <li><a href="https://www.backblaze.com/blog/reed-solomon/">Backblaze Open-sources Reed-Solomon Erasure Coding Source Code</a></li> <li><a href="https://web.stanford.edu/~skatti/pubs/usenix13-copysets.pdf">Copysets: Reducing the Frequency of Data Loss in Cloud Storage</a></li> <li><a href="https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf">Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution</a></li> <li><a href="https://www.usenix.org/system/files/fast21-pan.pdf">Facebook’s Tectonic Filesystem:Efficiency from Exascale</a></li> <li><a href="https://martin.kleppmann.com/2017/01/26/data-loss-in-large-clusters.html">The probability of data loss in large clusters</a></li> </ul>

2025/8/19

Inside HRT’s Python Fork: Leveraging PEP 690 for Faster Imports

<h4 class="wp-block-heading" id="h-python-hrt">Python @ HRT</h4> At Hudson River Trading (HRT), we’ve found that centralizing our codebase facilitates cross-team collaboration and rapid deployment of new projects. Therefore, the majority of our software development takes place in a monorepo, and our Python ecosystem is set up such that internal modules are importable everywhere. Unfortunately, the convenience of this arrangement has led to a conundrum: a vast proliferation of imports. In Python, imports occur at runtime. For each imported name, the interpreter must find, load, and evaluate the contents of a corresponding module. This process gets dramatically slower for large modules, modules on distributed file systems, modules with slow side-effects (code that runs during evaluation), modules with many transitive imports, and C/C++ extension modules with many library dependencies. Most of our internal modules fall into one or more of these categories. Thus, as the sheer number of imports has increased, so too has their cumulative runtime overhead. For users, this has surfaced as scripts starting tens of seconds later, notebooks taking minutes to load, and even the simplest distributed jobs spending a substantial portion of their runtime on imports. And yet, most of our scripts and modules <a href="https://dl.acm.org/doi/10.1145/3660821" target="_blank" rel="noreferrer noopener">only refer to a few of the names defined by the modules they import</a>. Applied recursively, this suggests that only a fraction of our imports are actually used at runtime. Is there some way we can avoid paying the cost of everything else? <h4 class="wp-block-heading" id="h-lazy-imports">Lazy Imports</h4> Lazy imports are a feature we borrowed from <a href="https://github.com/facebookincubator/cinder">Cinder</a>, Meta’s performance-oriented fork of CPython. The idea is to defer the resolution and evaluation of imported modules until they’re referenced, entirely bypassing imports that are never actually used at runtime. Cinder’s implementation of lazy imports relies on two core modifications: 1. Instead of immediately executing <code>import</code> statements, the interpreter assigns each imported name to a placeholder <code>LazyImport</code> object. This object persists everything needed to resolve and evaluate the import later on, i.e. the requisite module name and an optional <code>attribute</code> accessed by statements of the form <code>from module import attribute</code>. 2. When a <code>LazyImport</code> is retrieved from a <code>dict</code> (e.g. a name is referenced, incurring a <code>globals()</code> lookup) the interpreter completes the resolution and evaluation, transparently returning whatever was originally imported. For example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">from foo import bar # globals() now contains {"bar": LazyImport("foo", "bar")}. # the foo module has not yet been resolved or evaluated qux = bar() # in the evaluation of the above statement, bar is retrieved # from the globals() dict, causing foo to be imported and bar # to be reassigned to foo.bar</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M4.5 12.75l6 6 9-13.5"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>from foo import bar # globals() now contains {"bar": LazyImport("foo", "bar")}. # the foo module has not yet been resolved or evaluated qux = bar() # in the evaluation of the above statement, bar is retrieved # from the globals() dict, causing foo to be imported and bar # to be reassigned to foo.bar</code></pre></div> Of note, imports inside <code>try</code>/<code>except</code> and <code>with</code> blocks are never imported lazily. Both structures commonly handle import-related errors: for example, when detecting available or compatible modules. Therefore, exceptions from <code>import</code> statements inside these clauses are propagated immediately, rather than when they’re referenced, so exceptions can be appropriately handled. <h4 class="wp-block-heading" id="h-caveats">Caveats</h4> This implementation of lazy imports is not without its quirks. For example, optimizations around the storage of function locals make it too complicated to support lazy imports in functions. There’s also no way to make imports of the form <code>from module import *</code> lazy, as the names exported by a module cannot be determined until evaluation. More trivially, missing modules (and similarly, typos in import statements) are not apparent until they’re referenced. Another challenge presented by lazy imports is dealing with import side-effects. Some modules make externally-visible modifications when they’re evaluated. Examples include setting <code>sys.excepthook</code>, updating the root <code>logging</code> config, and adding <code>copyreg</code> callbacks. With lazy imports, these side-effects are invoked when the module is first referenced rather than when it’s imported, invalidating previously safe assumptions about global state. While imports of such modules can be made eager with a <code>try</code>/<code>except</code> or <code>with</code> block, it’s much less convenient to apply such changes to third-party packages. To mitigate this, we also ported Cinder’s lazy exclusion mechanism: a list of module names within which imports are always evaluated eagerly. We add all our third-party packages to this list (along with a couple internal packages that rely on the aforementioned patterns). <h4 class="wp-block-heading" id="h-migration">Migration</h4> A small, volunteer team of HRTers developed the initial prototype of lazy imports in Q1 2023 as part of a Surge—our annual, week-long, internal hackathon. In one week, they forked CPython 3.10, cherry-picked a handful of relevant commits from Cinder, and applied a few last fixes by hand to get everything working. Their presentation to the firm piqued widespread interest, and the project was greenlit for full-time commitment from Python@HRT (our dedicated, all-things-Python team). Over the following two quarters, we refined our implementation and began migrating the repository, starting with teams who were particularly excited to make the switch. Most cited slow script startup times or were already investigating runtime savings from manually inlining imports. The migration consisted of updating shebangs, refactoring test invocations, and fixing incompatibilities with lazy imports that generally fell into one of three categories: 1. Late or missing import side-effects. As we’ve established, lazy imports can cause import side-effects to be deferred or skipped entirely. Patterns we relied on in certain parts of the codebase, e.g. decorator registration, <code>__init_subclass__</code> hooks, and global overrides were no longer guaranteed to execute at startup—or at all. Debugging these incompatibilities was the most difficult part of the migration, as the result was often a silent change in behavior or a completely unrelated exception. Fortunately, the fix was usually as simple as wrapping the offending imports in a <code>try</code>/<code>except</code> or <code>with</code> block. 2. Import ordering. If multiple modules have conflicting side-effects, importers implicitly depend on the order in which those modules are evaluated. For example, in our codebase, some applications unintentionally needed a particular <code>sys.excepthook</code> to take precedence, while others accidentally required a precise sequence of logging configurations to produce the expected output. With lazy imports, no particular import ordering is guaranteed, so much of the migration was spent specifying these dependencies through <code>try</code>/<code>except</code> blocks, lazy exclusion, and explicit side-effect invocation where possible. 3. Implicit transitive imports. Regardless of how a submodule is imported, it’s always assigned as an attribute of its parent. As a result, programs can come to depend on submodules of their imports without directly importing them. For example, suppose I import packages <code>foo</code> and <code>bar</code>, and suppose that internally, <code>foo</code> imports <code>bar.baz</code>. Normally, <code>bar.baz</code> will automatically be imported and accessible in my code, even though I never imported it myself. With lazy imports, the <code>bar.baz</code> will only be accessible if <code>foo</code> is referenced, which isn’t guaranteed. Fixing this class of issues was much simpler, as the resulting <code>ImportErrors</code> were easy to spot and address. By Q2 2024, everyone independently interested in lazy imports had made the switch, bringing us to roughly 50% lazy compatibility across the monorepo. Around the same time, we started planning our upgrade to Python 3.12. We decided it would be minimally disruptive to make our 3.12 runtime lazy by default, finishing the lazy migration alongside the version upgrade rather than migrating the remainder of the firm twice. By Q3 2024, we had ported our implementation to CPython 3.12, and as of Q2 2025, we are lazy by default across the firm! <h4 class="wp-block-heading" id="h-the-benefits-of-laziness">The Benefits of Laziness</h4> Lazy imports have positively affected nearly all Python users at the firm. We attribute this to three key improvements. For command line tools, lazy imports have dramatically improved the “time to do something useful”, e.g. set up the environment, validate options, or even just show <code>--help</code>. For tools built around our largest internal packages, imports alone could cost users several minutes, severely disrupting their work. This could be especially painful if the script exited with an error shortly after starting, requiring yet another few minutes of waiting. By minimizing startup overhead, we tightened the feedback loop for CLIs across the board, helping researchers and engineers stay focused on sequential tasks. We observed similar benefits in Jupyter notebook workflows. HRT researchers often pick and choose functionality from a wide array of internal libraries. To make this process more convenient, they will often start interactive sessions by importing everything they might want to use in advance. Unfortunately, in certain side-effect heavy contexts, this can take 10 to 15 minutes. Eliminating this overhead by skipping or deferring imports has greatly reduced the friction of starting (or restarting) notebooks. Lastly, we found that lazy imports significantly improved the efficiency of our distributed compute framework. Many of our workloads involve coordinating large batches of computationally-intensive (but relatively short-lived) jobs. Distributed work is carefully prepared by the main application, typically amounting to a parameterized function. Consequently, each job tends to use very few of its application’s imports—maybe just a couple numeric libraries and a small subset of team-specific modules. Switching to lazy imports has eliminated time spent loading unused dependencies, which in extreme cases could take longer than the job itself! <h4 class="wp-block-heading" id="h-the-future">The __future__</h4> HRT has fully committed to lazy imports. Despite the maintenance and debugging costs associated with maintaining our own fork of the interpreter, we feel strongly that this feature has tangibly improved our Python ecosystem. Nonetheless, while laziness as an interpreter-wide feature works for us, we support the Steering Council in their <a href="https://discuss.python.org/t/pep-690-lazy-imports-again/19661/26">rejection</a> of <a href="https://peps.python.org/pep-0690/">PEP 690</a>—the implicit lazy imports are not a good fit for upstream due to the same, subtle bugs we encountered during our migration. However, as time permits, we hope to propose a revised lazy imports PEP that introduces an explicit <code>lazy</code> keyword, e.g. <code>lazy import foo</code> or <code>lazy from foo import bar</code>. This approach will satisfy migration and compatibility concerns, allow users to opt-in gradually, and enable all Python users to reap the speed benefits of lazy imports in a safe way.

2025/8/8

Intern Spotlight: HRT AI Labs Project

<a href="https://en.wikipedia.org/wiki/Diffusion_model">Diffusion models</a> are a widely popular model class for non-autoregressive generative modeling, yielding spectacular results in problem settings such as image generation. More recently, their application has been extended beyond computer vision into generative problems in areas such as NLP. In this project, we will explore their applicability to generate sequences of stock returns. Beyond generating random walks that capture key characteristics exhibited by stock returns (e.g. fat tails), we study a variety of problem settings to showcase diffusion models’ ability to model stock returns, focusing particularly on joint and conditional modeling. A key challenge persists in defining meaningful evaluation metrics to benchmark our models’ capabilities. In this way, we aim to demonstrate that our models capture ever more intricate aspects of the underlying return distributions. A natural question is whether we can generate sample returns conditioned on a given stock like AAPL or GOOG. We can train a model on returns with one-hot encodings for the stock, but a drawback of this approach is that we would need to retrain every time we want to generate samples for a new stock. Instead, in the same way that diffusion models condition on textual prompts by embedding them into a latent space, we can think of market metadata, like volume traded or historical volatility, as the latent features of a stock and train our model accordingly. This allows us to generate a wide variety of samples, even illiquid stocks with sparse returns or highly volatile stocks, without explicitly training on every stock. Plus, we can then finetune this large model for specific tasks. <figure class="wp-block-image alignwide"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXe3OpcooNbNpDOOcs3dPBXSoTfT-dWudfHe88db2irHOXxbocRxoVRqibU0-Vs0I1_URJIc0waO_8MtGtfyQ6PjLCGA7r4-fmV3iqo25iFotxmH4K7tGqIr4lkyTXQAj27qWDpJRptn__cPVJuV7B9cDDug?key=fU_7wZxvC8qkd40aLZ-Z1w" alt="" referrerpolicy="no-referrer"></figure> Another problem we think about is modeling cross-sectional returns across different stocks. For instance, we would expect some correlation between AAPL returns and similar tech stocks like GOOG; can a diffusion model generate these jointly? In image generation, diffusion models learn the joint distribution of red, green, and blue channels, so a natural way to model cross-sectional returns is to train a model on samples whose channels are returns for each stock. <figure class="wp-block-image alignwide"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXcrHGq_uihI0N3rQb6rwThkNF5HO0t5QyKygxCRbsD1l4feU-FngrKWXeUljbVIToAM4D19boR-ElRT6V38Msv9fpHw0QrTyfVEbulwYS8-qyjpDTEtX9pyyDoLrz0X9uCNwXyUeW2b5Jqr2onr0fqeZNvl?key=fU_7wZxvC8qkd40aLZ-Z1w" alt="" referrerpolicy="no-referrer"></figure> We can also try to generate returns conditioned on another return trajectory; if we are given sample GOOG returns, how do corresponding AAPL returns look? Given an image with missing regions, a diffusion model can <a href="https://arxiv.org/abs/2201.09865">inpaint</a> these regions with new, semantically reasonable content by first adding noise, then sequentially denoising the image and resampling from the known regions. But what if we masked entire channels? In our use case, we mask the channel for one stock (e.g. AAPL) and use our cross-sectional model to inpaint these channels conditioned on the other stock (e.g. GOOG). This works even without retraining the model for inpainting tasks! <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXe4Vm4EffDxzqLzHU1oCxxF2YPyxuXcjP5b1QTaAD1SAG584HRqnRpSdliOQfknXCEdy51_UbG2ocZRDuUWgihezffFqQBkCMkohfW3hAOIcJf1i7oY_QHZyNIS39WUvkl1WK0CyZu_ILDPTeZSZbua7Wc0?key=fU_7wZxvC8qkd40aLZ-Z1w" alt="" style="width:800px" referrerpolicy="no-referrer"></figure> While it is nice that these models can generate a variety of novel samples, how can we evaluate whether these samples are reasonably in-distribution? We can sanity-check by asking the model to generate many samples, then measure their fidelity to distributions of some aggregate metrics. For instance, returns of liquid stocks tend to be more fat-tailed than normal distributions, and daily volatility tends to be right-skewed. We’d also expect that “market signals” present in the original data would also be present in our synthetic data. For instance, returns are autocorrelated—and when predicting the magnitude of returns, we see that returns from a minute ago hold more predictive power than, say, returns from ten minutes earlier. <figure class="wp-block-image alignwide"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXf6F_lXFb1eAKeYiaAq-AG1ZyW1Rg0lNpVMJmxyVvnlzQGu-vwWzyO3A0eVH2GfiiHIfIQ5ffQ-icCdBmuVQnvsDfVhwnqPLbx7uA0AqWHxi88qK1gaCpnTCVW24h3Qj8AsNBHXye4rKPLAE5OQn0Oh3x2t?key=fU_7wZxvC8qkd40aLZ-Z1w" alt="" referrerpolicy="no-referrer"></figure> <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXdbH1-D1LrVaMTJG363qz8c4o0SPSuuCoWnGyqfFg5wHry3nfR-rpaCzvLlD-vmSIcRN8Iwy2b0gtzJ0yUBhp1FK3xlcFMRs1KV3X3gLF0PNcq2d1Nmz_l-kDe8SSEEm0yayruBS0e-n_o_dT16BVq7fk5A?key=fU_7wZxvC8qkd40aLZ-Z1w" alt="" style="width:800px" referrerpolicy="no-referrer"></figure> Ultimately, we found that diffusion models could be potentially powerful tools in generating synthetic time-series data like stock returns. As non-autoregressive models, diffusion models do not seem to struggle with degenerative issues like mode collapse, which is an issue we have seen in autoregressive models like transformers (<a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-hrt-ai-labs-summer-projects/">which we’ve tried before</a>!) or GANs.

2025/6/20

Intern Spotlight: Software Engineering Summer Projects

<h4 class="wp-block-heading" id="h-introduction">Introduction</h4> Each summer, HRT’s Software Engineering interns work with mentors, teams, and longtime HRTers on independent programming projects that tackle the most interesting problems in the world of automated trading, making a notable impact on our day-to-day operations. This particular intern spotlight series highlights the work of four of our 2024 Software Engineering interns, all of whom will be joining us full time in the not-too-distant future: <ul class="wp-block-list"> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#ParrotPneumaticTubing">Parrot Pneumatic Tubing by Alex Yi</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#DataService">Building a High Performance NIC-to-NVME Data Service by Alex Burton</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#SignalStreamCompression">Signal Stream Compression by Charlotte Wang</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#NonCOD">Non-COD OpenOrderServer by Liam Patterson</a></li> </ul> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-text-color has-black-color has-alpha-channel-opacity has-black-background-color has-background is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="ParrotPneumaticTubing">Parrot Pneumatic Tubing by Alex Yi</h2> <hr class="wp-block-separator has-text-color has-alpha-channel-opacity has-background is-style-wide" style="background-color:#eeeff8;color:#eeeff8"> <h3 class="wp-block-heading" id="h-it-s-like-a-drive-through-bank">It’s like a drive-through bank.</h3> <h4 class="wp-block-heading" id="h-background">Background</h4> At HRT we have a ton of internal trading APIs written in C++ which give us precise and efficient control over our systems. However, at times there is a desire to write succinct high-level code to express a new trade. For that, we have a system called Parrot (née <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-software-engineering-projects/#h-toucanserver">ToucanServer</a>). <figure class="wp-block-image aligncenter size-full is-resized is-style-default"><img loading="lazy" decoding="async" width="623" height="410" src="https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image1.png" alt="" class="wp-image-395660" style="width:700px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image1.png 623w, https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image1-480x316.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 623px, 100vw" referrerpolicy="no-referrer"></figure> The client-server design’s flexible, so we’re not tied to one client implementation, but client-server chatter uses gRPC under the hood, which incurs the overhead of a HTTP call per request (per order sent). When you also consider the extra network hop and the blocking behavior of the gRPC client, sending orders in a loop becomes unacceptably slow.  <h4 class="wp-block-heading">What To Do?</h4> gRPC is implemented over HTTP/2, which is great for scalability but not bandwidth: each (possibly small) individual order pays the cost of HTTP headers, burning through bandwidth. Instead we can send the headers just once! gRPC has a streaming protocol, allowing us to open just one long-lived request, which can then be used to pass multiple messages. Patching Parrot to do this was my project! <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="507" height="387" src="https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image2.png" alt="" class="wp-image-395661" style="width:700px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image2.png 507w, https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image2-480x366.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 507px, 100vw" referrerpolicy="no-referrer"></figure> To send a request with this new API, you wrap it into a stream-ready request and push it into the long-lived stream. The server will then unwrap it and dispatch it to the appropriate response handler and finally send a response back the way it came. Instead of trudging to the counter, waiting for the teller, and then trudging back, you get that glorious “FOOMP” as your check zooms away in a pneumatic tube. Ergonomics! <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="507" height="502" src="https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image3.png" alt="" class="wp-image-395662" style="width:700px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image3.png 507w, https://www.hudsonrivertrading.com/wp-content/uploads/2025/06/image3-480x475.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 507px, 100vw" referrerpolicy="no-referrer"></figure> To facilitate this, each stream call comes with a “clientUID” – a unique identifier that allows outgoing responses to be matched with incoming requests. Ultimately the server-side retooling was pretty minimal and mostly consisted of keeping track of clientUIDs and changing each handler to treat stream and normal calls equivalently. Unfortunately things were messier on the client side! Something has to handle the long-lived stream and responses might arrive out-of-order. So, I reimplemented the core of the client to use asynchronous operations under <a href="https://trio.readthedocs.io/en/stable/index.html">Trio</a> to handle all the concurrency, while keeping the same API but using async functions instead. This also allows for sending multiple orders at once (with something like asyncio.gather) should the need arise! <h4 class="wp-block-heading">Impact</h4> The ability to send many concurrent orders without having to muck around with threads is fantastic. But most importantly it all has been done unobtrusively, so Parrot devs can add more features with minimal hassle, and clients can reap the benefits just by sprinkling “awaits” into their existing code. Win-win! <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-text-color has-black-color has-alpha-channel-opacity has-black-background-color has-background is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="DataService">Building a High Performance NIC-to-NVME Data Service by Alex Burton</h2> <hr class="wp-block-separator has-text-color has-alpha-channel-opacity has-background is-style-wide" style="background-color:#eeeff8;color:#eeeff8"> HRT’s research environment has ever-increasing demands for high-throughput, low-latency distributed data storage. We have been prototyping several new storage systems, one of which is a design inspired by the Pangu <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#Bibliography">[1]</a> filesystem. <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXehqnFLb8NhBjBhUO2j3YUCz9duD-Nf6W4uhzpoBt8irJvZWdE3bikt46lCdfQBNnUCEfdPh0L7NbkC2D_X5zVX2dbuqGWOQwzwFR1enHAEgN2UQ0igU2C8ChWtzWGGKKin4hLRdMwC4ps_DBA12sWnWjqF?key=kQJwziDQ4HQ3rlZ8s9X6rA" alt="" style="width:500px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">Figure 1: Very high level architectural diagram of a distributed filesystem, eg. Pangu. The end user of the filesystem runs a client on their machine, which talks to metadata servers and data servers over the network. The data path is circled in blue.</figcaption></figure> Distributed file systems involve many different parts. In order to find something interesting for roughly 10 weeks of work, we focused on the data path between a client and a data server. <h4 class="wp-block-heading">The Data Path</h4> A data server provides a simple API over the network: a key-value store where entries can be created and deleted, and where values can be read from or appended to. Our first task was implementing this API locally. <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXeH8saI4PZ0YNAHgTdz-qRA5txxeVozqxLlgrt2i58LZpJOvZN_9BOHmm6Mvv5xVtzF1TVFNSCEjlSoTHFYgn55RXlnEn7stFyd9W2QuNjZsRA1HR_Duv3KRkQz9S6kODr_xIq19fJHWYE62CAy1_Ip_7CU?key=kQJwziDQ4HQ3rlZ8s9X6rA" alt="" style="width:500px" referrerpolicy="no-referrer"></figure> Figure 2: Magnified view of the data path. The data server (1) implements the key-value API locally, and (2) exposes the API over the network. <h4 class="wp-block-heading">Local Key-Value Storage</h4> As our baseline, we used a simple implementation on top of ext4 files using the trusty posix API to pack key/value tuples into files. Even when using multiple cores, this approach only yielded 1700 MB/s of multithreaded append throughput on an NVMe drive capable of 3800 MB/s write throughput. To obtain better performance, we decided to remove as many layers of abstraction as we could and interface directly with the NVMe device. To do this, we used Intel’s Storage Plane Development Kit (SPDK) <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#Bibliography">[3]</a> . We divided the drive into fixed-size write units. If the write unit size is large enough–around 128 KB on our test drive–we can hit the max raw write throughput of the drive with a single core by issuing random writes of this size, one after another. To avoid the overhead of journaling, we include a header in each write unit to allow each write to be self-identifying. To support fast reads, we store an in-memory map containing metadata for every key. If the server crashes, we can rebuild this map by scanning the disk and looking at the headers of each write unit. In the future, we would divide the drive into extents (fixed-size non-overlapping regions), checkpoint metadata, and limit recovery scanning to only extents that weren’t covered by the checkpoint. <h4 class="wp-block-heading">Key-Value Storage Over the Network</h4> At this point, our performance with a local workload was nearly matching the raw disk throughput, reaching 3600 MB/s of appends to random keys. To allow for workloads over the network, a simple solution would be to spin up an RPC server on top of the Linux TCP stack. However, Linux has relatively poor throughput and tail latency for networking, as suggested in the Luna paper <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#Bibliography">[4]</a>. The major culprits are unnecessary copies and scheduling overhead <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#Bibliography">[2, 4]</a>. To bypass these, we used the Data Plane Development Kit (DPDK) <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-software-engineering-summer-projects/#Bibliography">[5]</a> to directly interact with the NIC, allowing us to write zero-copy run-to-completion packet processing code. From here, we designed and implemented a simple RPC protocol (over UDP) on top of the key-value API. Since the maximum ethernet packet size is relatively small, each append RPC call needs to be split into multiple packets. Fortunately, NVMe drives support (restricted) scatter-gather lists for host memory buffers, so we can still perform zero-copy NVMe writes when the data to be written is scattered across many packets. Finally, after implementing the RPC protocol, we wrote a client load simulator. From benchmarking, we observed that the server was easily able to handle the incoming packets and issue appends at the speed the client could send them (around 2.6 GB/s–approaching the ethernet link capacity). Thus, we concluded interacting with the hardware directly in user-space unlocks much greater throughput with lower CPU usage. Bibliography: 1. <a href="https://www.usenix.org/system/files/fast23-li-qiang_more.pdf">https://www.usenix.org/system/files/fast23-li-qiang_more.pdf</a> 2. <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44271.pdf">https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44271.pdf</a> 3. <a href="https://spdk.io/">https://spdk.io/</a> 4. <a href="https://www.usenix.org/system/files/atc23-zhu-lingjun.pdf">https://www.usenix.org/system/files/atc23-zhu-lingjun.pdf</a> 5. <a href="https://www.dpdk.org/">https://www.dpdk.org/</a> Notes: <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-software-engineering/">https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-software-engineering/</a> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-text-color has-black-color has-alpha-channel-opacity has-black-background-color has-background is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="SignalStreamCompression">Signal Stream Compression by Charlotte Wang</h2> <hr class="wp-block-separator has-text-color has-alpha-channel-opacity has-background is-style-wide" style="background-color:#eeeff8;color:#eeeff8"> Market data enters our trading system as a stream of irregularly timed events corresponding to changes in the financial market: an order is placed, a trade is reported, a quote is published. In general, it’s useful to have such data re-sampled to be reported at regular intervals. We could then answer questions like: what was the mid price of AAPL at 10:00? In this system we refer to this re-sampled data as a “signal”, which is then transmitted to live-trading machines or saved to disk for future analysis. The efficient streaming of signals across networks and disks is thus an important problem at HRT. The focus of my project is to improve the compression of these signal streams.  <h4 class="wp-block-heading">Data Characteristics</h4> At each regular point in time, we need to transmit a row of sometimes dozens, sometimes thousands of signals. Because each signal is a time-series formed from irregularly-timed events, each metric changes very little, if at all, from one point in time (timestamp) to the next. This allows us to perform high levels of compression. Previously, rows were compressed by streaming the difference from the preceding row’s values. However, this only compresses identical values, and does not compress any values that have changed from row to row, even values with miniscule differences. Another inefficiency is that each entry includes an index of the changed value, meaning that the size of entry is dependent on the number of signals being transmitted per row.  <h4 class="wp-block-heading">The Solution </h4> In order to compress values with minuscule differences, we perform “XOR Compression” on these values, which are represented as a series of doubles.  <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXdOwCCnoUL-THxvFmn3G3EdelRhA7WbW3Tf6MUkwBiKN7E0Ja28M07cO7U6QYKSbLOTcMSVjswA-RHGNftB9ZL9Gm6rXPLSb8cCipqmcRUu902l9AHi_86s6VCLOTtiNZjKgOhe9KizwYEXeb6KmV3Ud_UL?key=APQJi8j-BdLyRGsp25QX3Q" alt="" style="width:800px" referrerpolicy="no-referrer"></figure> These bits are roughly transformed into a real number via the formula: (-1)sign* fraction*2exponent. Combined with the relatively stable magnitudes and precision of the time-series, we have that for a signal: <ul class="wp-block-list"> <li>The sign and exponent bits for should be unchanged through time</li> <li>Most data does not use the full fraction for precision, leaving the latter bits zero and unchanged through time. </li> </ul> Thus, the XOR between signals for adjacent times has lots of leading and trailing zeros that we can leave out (see scheme below).  <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXdZsuUGbqiWvJH8bU6A71b-C3yKUVkvCLRffYA0ddxM8tyuJ52G6_AfDWaMGamB8CoUF0JQVHOizTiogYVOPkmyMfGBH2p_FxPPCVSJMg6C72pANLEMO2MxL3S25hf8aN6C0iiLWObXJQqU5Db5GeyyoXwW?key=APQJi8j-BdLyRGsp25QX3Q" alt="" style="width:800px" referrerpolicy="no-referrer"></figure> This XOR compression scheme eliminates the need for indices of changed values. By consolidating multiple logical values into contiguous bytes, a single ‘0’ bit can encode “no change,” allowing us to simply record the updates for each value. <h4 class="wp-block-heading">Results</h4> <div class="wp-block-group is-content-justification-center is-nowrap is-layout-flex wp-container-core-group-is-layout-d05cb3ef wp-block-group-is-layout-flex"> <figure class="wp-block-image aligncenter"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXcNdZm2e8fP6QxKih18mGBzU8-VHFbtjn9ecCZn4Q1yM8gvvCdIswMBka-k3sXEvnrJHcmcPsvY8qJ6j8uYYuyRnC16N1OlkdNB1enhcQjKaV5_CeuuHBbJUSd2tGG8BZAgDeJWyloA-yzhJUFobTkRUqhD?key=APQJi8j-BdLyRGsp25QX3Q" alt="" style="object-fit:cover" referrerpolicy="no-referrer"></figure> <figure class="wp-block-image"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXfGWEeIHngTuCv3jZ86AYJvH4K70CXyqwl3c5Xxzr4n49ETEZSC_GNUevlnls3PL5Q-rKf479NK8zLI2llhLqg3LuZ9baYWv6Bjo6S154tr8V2sVfuMmcYoGtU4VitJMOTyQCU3nZWtZJDqyuRDYSwiFkc6?key=APQJi8j-BdLyRGsp25QX3Q" alt="" referrerpolicy="no-referrer"></figure> </div> As expected, many of the compressed values need only 1 bit because there is no change. Most values are compressed to 20 – 50 bits, grouped by different precision classes. I tested this new compression strategy on a selection of 5 different sample data workloads. Each 64-bit double was compressed to an average of ~10 bits, half of the ~20 bits required from the previous signal diff format!  <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-text-color has-black-color has-alpha-channel-opacity has-black-background-color has-background is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="NonCOD">Open Order Server for non-Cancel-on-Disconnect orders by Liam Patterson</h2> <hr class="wp-block-separator has-text-color has-alpha-channel-opacity has-background is-style-wide" style="background-color:#eeeff8;color:#eeeff8"> At HRT we generally want our open orders to be canceled if we lose connection to the market in order to reduce risk. As a result orders are typically configured to be cancel-on-disconnect (COD). However, as HRT expands into new markets and styles of trading, there’s an increasing need for orders that don’t have this behavior. Canceling auction orders, for example, is considered risk-increasing since we expect those orders to be filled. <h4 class="wp-block-heading" id="h-problem">Problem</h4> In order to understand our exposure on the market, we need to know our currently open orders. There’s limited infrastructure in place to monitor open non cancel-on-disconnect orders. The source of truth for open orders is our order entry server, called Trip, which is the process that connects to exchanges and sends orders. As we trade Trip multicasts events and writes these events to a log file. We have two existing options for discovering our open orders. The first is to read the massive log output sequentially, which is a slow process. The second is to directly ask Trip for its open orders, but this interrupts regular trading, and more importantly doesn’t work if Trip crashes. <h4 class="wp-block-heading" id="h-solution">Solution</h4> For this internship, I developed the Open Order Server (OOS): a process which subscribes to Trip multicast data and keeps track of non-COD open order state. When a process wants to query its open non-COD orders, it asks the OOS rather than the Trip. The server maintains the set of open orders for trips that go offline (either intentionally or due to a crash). During normal operation, the OOS tracks order state by listening to streamed multicast data. On startup the OOS first needs to initialize its state from Trip if the latter is already running. It does this by requesting open orders from Trip, which is similar to the existing workflow. However, if OOS starts while Trip is down, then the server reads the log file instead. As soon as a trip comes online, the server will try to connect to it and request open orders directly. <h4 class="wp-block-heading" id="h-oos-trip-state-diagram">OOS Trip State Diagram:</h4> <figure class="wp-block-image aligncenter is-resized"><img decoding="async" src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXcqVyOkPHAxVKckD4mttTZShNdJmOYc2-bc1gRhaXCcPosgXof4NqikVALDQbb9PfQ1_7gRG695rhawpnVdnLKR-eZJjK3qrDrHe7-Mdr_MWBjOER33r78omqnW7O6pcjySy-iR_K8f1104TcqOMHqTkVrT?key=PTnF7dTWsUpTHmZS_q7QDg" alt="" style="width:700px" referrerpolicy="no-referrer"></figure> One technical challenge was to implement the smooth transition between the refresh and the live streaming data without any gaps. This transition was mostly handled by an earlier intern project, but I had to extend the existing logic to support refreshing all open orders as opposed to being limited to orders for a predetermined set of symbols. By the end of my internship the OOS was able to refresh state from trip order logs, listen to multicast order data, and respond to queries about open orders. Now we are able to quickly and reliably know our market exposure whether or not Trip is running, which enables HRT to explore new trading around non-COD orders.

2025/6/12

Optimising Compiler Performance: A Case For Devirtualisation

In the realm of high frequency trading, speed is critical. Programmers working on production systems at firms such as HRT must write code that is highly performant without compromising correctness. To achieve that, we should be aware of how compilers work, the ways they can optimise our code, and potential limitations. Let’s dive a bit deeper into one type of optimisation that compilers perform – devirtualisation. <h3 class="wp-block-heading" id="h-introduction-virtual-functions">Introduction – virtual functions</h3> In an inheritance hierarchy, derived classes often specialise some behaviour by overriding certain methods of their base class. This allows for polymorphism: a call made through a reference to the same base type (<code>Shape&</code>) can behave differently depending on the derived type of the object to which it refers (<code>Circle</code>, <code>Square</code>, etc). In C++, methods whose behaviour can be overridden in this way must be marked as virtual, which instructs the compiler to emit additional logic needed to resolve the proper implementation at runtime. Specifically, for each type with virtual functions, the compiler generates a virtual table (vtable) containing one function pointer for the appropriate implementation of each virtual method of that type. Each object of that type stores a pointer (v-pointer) to the vtable for its type, and virtual method calls are implemented by emitting code that loads the appropriate function pointer from the vtable and calls it. While virtual functions are an extremely useful feature, they come at a price. The v-pointers make objects larger (which impacts cache efficiency), the vtables take up additional cache space and introduce potential cache misses, the indirect calls can result in branch mispredictions, and the reduced compile-time knowledge prevents inlining and other optimisations. Fortunately, in many cases compilers can give us both flexibility and speed through devirtualisation — deducing which function is needed at compile time and producing machine code that calls it directly instead of going through the vtable. This post will explore when devirtualisation is possible, what the potential gains are, and how to write code for the compiler to optimise this way. <h3 class="wp-block-heading" id="h-what-virtual-functions-get-compiled-to">What virtual functions get compiled to</h3> To understand the performance impact of virtualisation, let’s start by examining what a virtual call gets compiled to. Here’s a simple example, in which we call a regular function: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">class Base { public: int foo(); }; int f(Base* b) { return b->foo() + 0xc0ffee; }</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>class Base { public: int foo(); }; int f(Base* b) { return b->foo() + 0xc0ffee; }</code></pre></div> To inspect what it compiles to, let’s look at the relevant part of the output of <code>objdump -dC -M intel -r</code> after compiling the code with gcc on x86_64: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">0000000000000000 <f(Base*)>: 0: 48 83 ec 08 sub rsp,0x8 4: e8 00 00 00 00 call 9 <f(Base*)+0x9> 5: R_X86_64_PLT32 Base::foo()-0x4 9: 48 83 c4 08 add rsp,0x8 d: 05 ee ff c0 00 add eax,0xc0ffee 12: c3 ret</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>0000000000000000 <f(Base*)>: 0: 48 83 ec 08 sub rsp,0x8 4: e8 00 00 00 00 call 9 <f(Base*)+0x9> 5: R_X86_64_PLT32 Base::foo()-0x4 9: 48 83 c4 08 add rsp,0x8 d: 05 ee ff c0 00 add eax,0xc0ffee 12: c3 ret</code></pre></div> Now, consider a call to a virtual function: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">class Base { public: virtual int foo() = 0; }; int f(Base* b) { return b->foo() + 0xc0ffee; }</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>class Base { public: virtual int foo() = 0; }; int f(Base* b) { return b->foo() + 0xc0ffee; }</code></pre></div> Function <code>f</code> in this case compiles to: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">0000000000000000 <f(Base*)>: 0: 48 83 ec 08 sub rsp,0x8 4: 48 8b 07 mov rax,QWORD PTR [rdi] 7: ff 10 call QWORD PTR [rax] 9: 48 83 c4 08 add rsp,0x8 d: 05 ee ff c0 00 add eax,0xc0ffee 12: c3 ret</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>0000000000000000 <f(Base*)>: 0: 48 83 ec 08 sub rsp,0x8 4: 48 8b 07 mov rax,QWORD PTR [rdi] 7: ff 10 call QWORD PTR [rax] 9: 48 83 c4 08 add rsp,0x8 d: 05 ee ff c0 00 add eax,0xc0ffee 12: c3 ret</code></pre></div> The register <code>rdi</code> initially holds the <code>b</code> argument, and since <code>Base::foo</code> is a virtual method, a lookup in the vtable is necessary. Instead of directly calling a function at a known address, we need two memory accesses: one to obtain the v-pointer and one to obtain the function address. Each memory load is an opportunity for an expensive cache miss. When the call does occur, it is calling an address that was loaded from memory rather than one written into the machine code, which is harder for the processor to predict. Finally, since the compiler doesn’t know what implementation will be invoked, it can’t perform optimisations that involve the context from which the function is called, such as constant propagation and inlining. Let’s take a look at a benchmark to see how a lack of this optimisation impacts performance. <a href="https://github.com/hudson-trading/hrtbeat/tree/master/devirtualisation" target="_blank" rel="noreferrer noopener">The benchmark </a>does a simple O(N3) matrix multiplication multiple times. In one case, the function is in the same compilation unit as <code>main</code>, so it can be inlined and constant-propagated; in the other case, it’s in another compilation unit and compiled separately. All functions are compiled with <code>-O3</code> and <code>-mavx</code>. The benchmarks were run with CPU pinning on a modern AMD processor — the first one averaged 19.157s per run, while the second one averaged 19.519s per run. This difference may seem small, but decreasing run time by 2% on the critical path is a big win. Further, the results are statistically significant; performing a t-test on them gives a p-value of 10-60. <h3 class="wp-block-heading" id="h-compilers-are-relatively-smart">Compilers are relatively smart</h3> In most real-life cases, the compiler tries to predict which code paths are most likely to be accessed. In certain situations, it can deduce which function needs to be called at compile time and call it directly instead of going through the vtable. Let’s dig into some cases where it’s possible. Modern compilers can devirtualise a call when they are able to prove that a particular object has a fixed type, such as when we directly create the object of a particular derived type and then call its virtual methods. Devirtualisation is even possible when we have a pointer to the base class, as long as the compiler is sure where the object was created and what type it has.<a href="https://www.hudsonrivertrading.com/hrtbeat/optimising-compiler-performance-a-case-for-devirtualisation/#05b7ee08-88f4-47eb-a7ec-3a9b8f430f9f" id="05b7ee08-88f4-47eb-a7ec-3a9b8f430f9f-link">1</a> However, this is often not enough in practice, where objects are passed around between different compilation units. To understand why splitting code between different compilation units matters, let’s look at how compilers optimise code. We’ll look closely at gcc, but the general ideas are shared among other implementations as well. In a traditional build workflow, code is first compiled into gcc’s IR, <a href="https://gcc.gnu.org/onlinedocs/gccint/GIMPLE.html" target="_blank" rel="noreferrer noopener">GIMPLE</a>. Then, the optimiser performs various passes through GIMPLE before producing an object file. Multiple object files then get linked into an executable. This approach makes a lot of sense from a programmer’s point of view – different parts of code are separated into modules and only modules that change need to be recompiled, so the feedback loop is shorter. However, from the compiler’s point of view, a lot of possible optimisations are missed as a result. There are <a href="https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html" target="_blank" rel="noreferrer noopener">two noteworthy compiler flags</a> that change this behaviour and can mitigate these problems. The first one is <code>-fwhole-program</code>. This flag is useful when optimising small programs that fit into one compilation unit – it tells the compiler that what it is compiling now is the whole program. As a result, it can assume that if nothing derives from a certain class in the current context, then calls to methods on objects of this class do not have to be dynamically resolved through the vtable.  Since there are no derived classes that could override the functions, it is safe to devirtualise them. However, fitting everything into one compilation unit can be too restrictive for well-factored programs. To help with that, there is another flag that can result in similar optimisations. This flag is <code>-flto</code>. LTO stands for link time optimisation. This flag allows the compiler to collectively treat multiple source files as a single compilation unit for the purpose of optimisation. It significantly changes gcc’s output, as it causes it to write GIMPLE to the resulting ELF object file instead of standard code sections. During linking, all the object files that were created with this flag can be optimised as a single GIMPLE unit, allowing for more aggressive devirtualisation, inlining, const propagation, and other optimisations. For example, functions from one file can be inlined into another one, which can result in propagation of more constants, devirtualisation of more methods, and so on. A noteworthy feature of LTO is that it allows for optimisation across multiple programming languages supported by gcc, as they all get compiled into GIMPLE first, optimised, and then compiled into assembly. Worth mentioning here is the <code>final</code> keyword. It is a way C++ allows programmers to hint to the compiler to better optimise the code, achieving similar results to <code>-fwhole-program</code> if done consistently. It is extremely useful in cases where inheritance is mainly used for sharing some base functionality between multiple classes and instances are kept as objects of their final derived types. For example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">class Base { public: virtual int foo(); virtual int bar(); }; class Derived1 : public Base { public: int foo() override; int bar() final; }; int test(Derived1* d1) { return d1->foo() + d1->bar(); }</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>class Base { public: virtual int foo(); virtual int bar(); }; class Derived1 : public Base { public: int foo() override; int bar() final; }; int test(Derived1* d1) { return d1->foo() + d1->bar(); }</code></pre></div> Function test compiles to: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">0000000000000000 <test(Derived1*)>: 0: 55 push rbp 1: 48 89 fd mov rbp,rdi 4: 53 push rbx 5: 48 83 ec 08 sub rsp,0x8 9: 48 8b 07 mov rax,QWORD PTR [rdi] c: ff 10 call QWORD PTR [rax] e: 48 89 ef mov rdi,rbp 11: 89 c3 mov ebx,eax 13: e8 00 00 00 00 call 18 <test(Derived1*)+0x18> 14: R_X86_64_PLT32 Derived1::bar()-0x4 18: 48 83 c4 08 add rsp,0x8 1c: 01 d8 add eax,ebx 1e: 5b pop rbx 1f: 5d pop rbp 20: c3 ret</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>0000000000000000 <test(Derived1*)>: 0: 55 push rbp 1: 48 89 fd mov rbp,rdi 4: 53 push rbx 5: 48 83 ec 08 sub rsp,0x8 9: 48 8b 07 mov rax,QWORD PTR [rdi] c: ff 10 call QWORD PTR [rax] e: 48 89 ef mov rdi,rbp 11: 89 c3 mov ebx,eax 13: e8 00 00 00 00 call 18 <test(Derived1*)+0x18> 14: R_X86_64_PLT32 Derived1::bar()-0x4 18: 48 83 c4 08 add rsp,0x8 1c: 01 d8 add eax,ebx 1e: 5b pop rbx 1f: 5d pop rbp 20: c3 ret</code></pre></div> We already know from the earlier function call investigation that the first call to foo() is virtual (since d1 could be something that derived from Derived1, not necessarily just Derived1). However, the second call to bar() is not virtual since the function is marked as final, so the compiler knows it’s safe to devirtualise. <h3 class="wp-block-heading" id="h-help-the-compiler-be-smart">Help the compiler be smart</h3> In cases where the compiler doesn’t have the ability to devirtualise on its own, it is possible to use a Curiously Recurring Template Pattern, a C++ idiom important enough to earn a <a href="https://en.cppreference.com/w/cpp/language/crtp" target="_blank" rel="noreferrer noopener">place on cppreference</a>. The basic premise of CRTP is for the derived class to inherit from the base class, templated by the derived class. This means that for every derived class, there is a separate instance of the base class, and instead of calling virtual methods on children, casting <code>this</code> to the derived class and calling normal members is possible. This way, static polymorphism can be implemented for the cases when the only calls to virtual methods are made from regular class methods. A simple example showing CRTP is: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">template <typename Derived> class Base { public: int foo() { return 0xdead0000 + static_cast<Derived*>(this)->bar(); } }; class Derived : public Base<Derived> { public: int bar() { return 0xbabe; } }; int test() { Derived d; return d.foo(); }</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>template <typename Derived> class Base { public: int foo() { return 0xdead0000 + static_cast<Derived*>(this)->bar(); } }; class Derived : public Base<Derived> { public: int bar() { return 0xbabe; } }; int test() { Derived d; return d.foo(); }</code></pre></div> Function test compiles (with -O3) into: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">0000000000000000 <test()>: 0: b8 be ba ad de mov eax,0xdeadbabe 5: c3 ret</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>0000000000000000 <test()>: 0: b8 be ba ad de mov eax,0xdeadbabe 5: c3 ret</code></pre></div> This shows the compiler was able to optimise out both the call to foo and to bar, and no virtual tables were involved. Though this method offers performance improvements, it can also significantly limit the way the derived class can be used and make the code more complicated to understand. While sharing similar code across multiple derived classes is possible, there is no real common base class between them. As a result, every functionality that is supposed to be shared between derived classes has to be a template, even if it’s not encapsulated in the base class.  C++23 offers an interesting alternative to CRTP – <a href="https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p0847r6.html" target="_blank" rel="noreferrer noopener">deducing this</a>. It allows for moving the <code>Derived</code> template parameter to <code>Self</code> template parameter for methods that actually need to call the derived class. This allows our previous example to be rewritten as something along the lines of: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">class Base { public: template <typename Self> int foo(this Self&& self) { return 0xdead0000 + self->bar(); } }; class Derived : public Base { public: int bar() { return 0xbabe; } }; int test() { Derived d; return d.foo(); }</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>class Base { public: template <typename Self> int foo(this Self&& self) { return 0xdead0000 + self->bar(); } }; class Derived : public Base { public: int bar() { return 0xbabe; } }; int test() { Derived d; return d.foo(); }</code></pre></div> An interesting feature of this proposal is that, contrary to CRTP, derived classes have a common base, making it possible to cast up to that class. However, since there is no dynamic dispatch, calling a method that is overridden in the derived class still calls this method from the base class, which is counterintuitive and different from the expected behaviour. Overall, this proposal offers an alternative to CRTP , and makes resulting code easier to understand — especially in cases of deep inheritance hierarchies, where double-derived classes derive from derived classes. <h3 class="wp-block-heading" id="h-more-musings-about-the-compiler-trying-to-be-smart-and-failing">More musings about the compiler trying to be smart and failing</h3> There is one more optimisation worth mentioning, though it can also have harmful effects. It is called speculative devirtualisation, and it works by checking if the vtable entry is equal to the address of one of the possible functions, the one deemed most likely to be the one that needs to be called. It only works if the compiler is relatively confident which function to check for; if the guess is correct, a direct call is made instead of a virtual one. This is a great solution if the compiler is able to guess correctly, as it allows that branch to be potentially inlined (which, as explored previously, brings enormous benefits) and it might help with the binary layout. However, some heuristics are easy to trigger accidentally, especially in a large codebase shared across many programmers. As an example, let’s say only one implementation is visible at compile time, for instance because it was placed in a header file rather than in a source file. The compiler likes to assume this is the most probable function,  but it may not be! It may just have been short enough to be reasonably placed in the source file! Or it may have been completely arbitrary! Using this implementation for speculative execution can result in very bad branch prediction, cache misses, and dumb inlining. While this can be fixed by making none of the implementations visible at compile time and using <code>-flto</code>, C++ programmers should still be aware of these consequences. To illustrate this interesting case, let’s look at a very simple example. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">struct Base { long long foo() { return bar() + 0xf01dab1e00000000LL; } virtual long long bar() = 0; }; struct Derived : public Base { long long bar() override { return 0x5ca1ab1e; } }; long long test(Base* b) { return b->foo(); }</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>struct Base { long long foo() { return bar() + 0xf01dab1e00000000LL; } virtual long long bar() = 0; }; struct Derived : public Base { long long bar() override { return 0x5ca1ab1e; } }; long long test(Base* b) { return b->foo(); }</code></pre></div> Function test compiles to: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><textarea class="code-block-pro-copy-button-textarea" aria-hidden="true" readonly="">0000000000000000 <test(Base*)>: 0: 48 8b 07 mov rax,QWORD PTR [rdi] 3: 48 8d 0d 00 00 00 00 lea rcx,[rip+0x0] # a <test(Base*)+0xa> 6: R_X86_64_PC32 Derived::bar()-0x4 a: 48 8b 10 mov rdx,QWORD PTR [rax] d: 48 39 ca cmp rdx,rcx 10: 75 0e jne 20 <test(Base*)+0x20> 12: 48 b8 1e ab a1 5c 1e movabs rax,0xf01dab1e5ca1ab1e 19: ab 1d f0 1c: c3 ret 1d: 0f 1f 00 nop DWORD PTR [rax] 20: 48 83 ec 08 sub rsp,0x8 24: ff d2 call rdx 26: 48 83 c4 08 add rsp,0x8 2a: 48 ba 00 00 00 00 1e movabs rdx,0xf01dab1e00000000 31: ab 1d f0 34: 48 01 d0 add rax,rdx 37: c3 ret</textarea><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>0000000000000000 <test(Base*)>: 0: 48 8b 07 mov rax,QWORD PTR [rdi] 3: 48 8d 0d 00 00 00 00 lea rcx,[rip+0x0] # a <test(Base*)+0xa> 6: R_X86_64_PC32 Derived::bar()-0x4 a: 48 8b 10 mov rdx,QWORD PTR [rax] d: 48 39 ca cmp rdx,rcx 10: 75 0e jne 20 <test(Base*)+0x20> 12: 48 b8 1e ab a1 5c 1e movabs rax,0xf01dab1e5ca1ab1e 19: ab 1d f0 1c: c3 ret 1d: 0f 1f 00 nop DWORD PTR [rax] 20: 48 83 ec 08 sub rsp,0x8 24: ff d2 call rdx 26: 48 83 c4 08 add rsp,0x8 2a: 48 ba 00 00 00 00 1e movabs rdx,0xf01dab1e00000000 31: ab 1d f0 34: 48 01 d0 add rax,rdx 37: c3 ret</code></pre></div> The compiler has no reason to believe that the provided implementation of function <code>bar</code> will be the one most commonly used, but as we can see in the assembly, it compares the address to the vtable (<code>QWORD PTR [rdi]</code>) with something (as we can guess, address of vtable of Derived), and if they are not equal, it does a jump. If they are equal, it simply returns <code>0xf01dab1e5ca1ab1e</code>. This layout of code shows the expectation here is that they will be equal, even though the compiler has no reason to believe it. <h3 class="wp-block-heading" id="h-summary">Summary</h3> Modern C++ compilers are smart and try to find the best solutions to make code faster. As C++ programmers, we should be aware of the way compilers were designed and why they behave the way that they do so we can write code that is likely to be optimised. This involves using certain patterns such as CRTP, but also avoiding certain shortfalls, such as making only one definition of a function visible. It also always makes sense to measure, measure, and measure some more the performance of the code, and occasionally look into what assembly was actually generated, to better understand what the compiler is doing and how to optimise performance of the resulting binaries. <style> .et_parallax_bg { background-position: center; } .broken_link, a.broken_link {text-decoration: none !important;"} </style> <ol class="wp-block-footnotes"><li id="05b7ee08-88f4-47eb-a7ec-3a9b8f430f9f">It is also possible to confuse the compiler with code where it is obvious what the type of a particular object is to a human reader: example <a href="https://quuxplusone.github.io/blog/2021/02/15/devirtualization/" target="_blank" rel="noreferrer noopener">here</a>. <a href="https://www.hudsonrivertrading.com/hrtbeat/optimising-compiler-performance-a-case-for-devirtualisation/#05b7ee08-88f4-47eb-a7ec-3a9b8f430f9f-link" aria-label="Jump to footnote reference 1">↩︎</a></li></ol>

2024/5/21

Building Robust Codebases with Python’s Type Annotations

Hudson River Trading’s (HRT’s) Python codebase is large* and constantly evolving. Millions of lines of Python reflect the work of hundreds of developers over the last decade. We trade in over 200 markets worldwide — including nearly all of the world’s electronic markets — so we need to regularly update our code to handle changing rules and regulations. * To learn how we handle large-scale code organization, see: <a href="https://www.hudsonrivertrading.com/hrtbeat/dependency-graph-python-codebase/">Building a Dependency Graph of Our Python Codebase</a>. Our codebase provides command-line interface (CLI) tools, graphical user interfaces*  (GUIs), and event-triggered processes that assist our traders, engineers, and operations personnel. This outer layer of our codebase is supported by an inner layer of shared business logic. Business logic is often more complicated than it appears: even a simple question like “what is the next business day for NASDAQ?” involves querying a database of market calendars (a database that requires regular maintenance). So, by centralizing this business logic into a single source of truth, we ensure that all the different systems in our codebase behave coherently. * To learn about our GUI design process, see: <a href="https://www.hudsonrivertrading.com/hrtbeat/optimizing-ux-ui-design-for-trading/">Optimizing UX/UI Design for Trading at HRT</a>. Even a small change to shared business logic can affect many systems, and we need to check that these systems won’t have issues with our change. It’s inefficient and error-prone for a human to manually verify that nothing is broken. Python’s type annotations significantly improve how quickly we can update and verify changes to shared business logic. Type annotations allow you to describe the type of data handled by your code. “Type checkers” are tools that reconcile your descriptions against how the code is actually being used. When we update shared business logic, we update the type annotations and use a type checker to identify any downstream systems that are affected. We also thoroughly document and test our codebase. But written documentation is not automatically synchronized with the underlying code, so maintaining documentation requires a high level of vigilance and is subject to human error. Additionally, automated testing is limited to the scenarios that we test for,* which means that novel uses of our shared business logic will be unverified until we add new tests. * Generative testing via <a href="https://hypothesis.works/">Hypothesis</a> is a promising new avenue that we are also exploring to address some of the deficiencies of more traditional automated testing. The rest of this article will explore: <ul class="wp-block-list"> <li>How type annotations work in Python</li> <li>The type checking tools that HRT uses</li> <li>An example of detecting downstream issues</li> <li>Some tips and tricks for using type annotations effectively</li> </ul> <h2 class="wp-block-heading" id="h-how-type-annotations-work-in-python">How type annotations work in Python</h2> <hr class="wp-block-separator has-text-color has-white-color has-alpha-channel-opacity has-white-background-color has-background is-style-wide"> Let’s look at an example of type annotations to see how they can be used to describe the shape of data. Here’s some type annotated Python code that computes the checksum digit of a <a href="https://en.wikipedia.org/wiki/CUSIP">CUSIP</a>: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">def cusip_checksum(cusip8: str) -> int: assert len(cusip8) == 8 chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#" charmap: dict[str, int] = { char: value for value, char in enumerate(chars, start=0) } total: int = 0 for idx, char in enumerate(cusip8, start=0): value: int = charmap[char] if (idx % 2) == 1: value *= 2 total += (value // 10) + (value % 10) return (10 - total % 10) % 10</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>def cusip_checksum(cusip8: str) -> int: assert len(cusip8) == 8 chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#" charmap: dict[str, int] = { char: value for value, char in enumerate(chars, start=0) } total: int = 0 for idx, char in enumerate(cusip8, start=0): value: int = charmap[char] if (idx % 2) == 1: value *= 2 total += (value // 10) + (value % 10) return (10 - total % 10) % 10</code></pre></div> Here’s what the type annotations tell us: <ul class="wp-block-list"> <li><code>cusip_checksum()</code> is a function that takes a string as input and returns an integer as output.</li> <li><code>chars</code> is a string.</li> <li><code>charmap</code> is a dictionary with string keys and integer values.</li> <li><code>total</code> and <code>value</code> are integers.</li> </ul> You might notice that the type annotations don’t describe everything about the data. For example: <ul class="wp-block-list"> <li>The input to <code>cusip_checksum()</code> should be an 8-character string.</li> <li>The output of <code>cusip_checksum()</code> should be an integer in the range 0 to 9.</li> <li><code>charmap</code> is a dictionary whose keys are single-character strings.</li> </ul> Even though Python doesn’t have support for <a href="https://en.wikipedia.org/wiki/Refinement_type">refinement types</a> (yet), you can address this in several ways. First of all, this may not be an issue for you: the level of granularity provided just by <code>str</code> and <code>int</code> may be sufficient for your code. Additionally, Python’s <a href="https://docs.python.org/3/library/typing.html">typing</a> module provides tools for more advanced type constraints, like <code>NewType()</code>, <code>Literal[]</code>, and <code>TypeGuard[]</code>. Type annotations were added in Python 3.5 via a Python Enhancement Proposal (PEP):  <a href="https://peps.python.org/pep-0484/">PEP 484 – Type Hints</a>. Since then, there have been a number of PEPs to improve type annotations further: <ul class="wp-block-list"> <li><a href="https://peps.python.org/pep-0482/">PEP 482 – Literature Overview for Type Hints</a></li> <li><a href="https://peps.python.org/pep-0483/">PEP 483 – The Theory of Type Hints</a></li> <li><a href="https://peps.python.org/pep-0484/">PEP 484 – Type Hints</a></li> <li><a href="https://peps.python.org/pep-0544/">PEP 544 – Protocols: Structural subtyping (static duck typing)</a></li> <li><a href="https://peps.python.org/pep-0560/">PEP 560 – Core support for typing module and generic types</a></li> <li><a href="https://peps.python.org/pep-0563/">PEP 563 – Postponed Evaluation of Annotations</a></li> <li><a href="https://peps.python.org/pep-0585/">PEP 585 – Type Hinting Generics In Standard Collections</a></li> <li><a href="https://peps.python.org/pep-0586/">PEP 586 – Literal Types</a></li> <li><a href="https://peps.python.org/pep-0589/">PEP 589 – TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys</a></li> <li><a href="https://peps.python.org/pep-0591/">PEP 591 – Adding a final qualifier to typing</a></li> <li><a href="https://peps.python.org/pep-0593/">PEP 593 – Flexible function and variable annotations</a></li> <li><a href="https://peps.python.org/pep-0604/">PEP 604 – Allow writing union types as X | Y</a></li> <li><a href="https://peps.python.org/pep-0612/">PEP 612 – Parameter Specification Variables</a></li> <li><a href="https://peps.python.org/pep-0613/">PEP 613 – Explicit Type Aliases</a></li> <li><a href="https://peps.python.org/pep-0646/">PEP 646 – Variadic Generics</a></li> <li><a href="https://peps.python.org/pep-0647/">PEP 647 – User-Defined Type Guards</a></li> <li><a href="https://peps.python.org/pep-0649/">PEP 649 – Deferred Evaluation Of Annotations Using Descriptors</a></li> <li><a href="https://peps.python.org/pep-0655/">PEP 655 – Marking individual TypedDict items as required or potentially-missing</a></li> <li><a href="https://peps.python.org/pep-0673/">PEP 673 – Self Type</a></li> <li><a href="https://peps.python.org/pep-0675/">PEP 675 – Arbitrary Literal String Type</a></li> <li><a href="https://peps.python.org/pep-0692/">PEP 692 – Using TypedDict for more precise **kwargs typing</a></li> <li><a href="https://peps.python.org/pep-0695/">PEP 695 – Type Parameter Syntax</a></li> <li><a href="https://peps.python.org/pep-0696/">PEP 696 – Type defaults for TypeVarLikes</a></li> <li><a href="https://peps.python.org/pep-0698/">PEP 698 – Override Decorator for Static Typing</a></li> <li><a href="https://peps.python.org/pep-0702/">PEP 702 – Marking deprecations using the type system</a></li> <li><a href="https://peps.python.org/pep-0705/">PEP 705 – TypedMapping: Type Hints for Mappings with a Fixed Set of Keys</a></li> </ul> <h2 class="wp-block-heading" id="h-the-type-checking-tools-that-hrt-uses">The type checking tools that HRT uses</h2> <hr class="wp-block-separator has-text-color has-white-color has-alpha-channel-opacity has-white-background-color has-background is-style-wide"> HRT uses <a href="https://www.mypy-lang.org/">mypy</a> to analyze our Python type annotations. Mypy works by analyzing the type annotations in one or more Python files and determining if there are any issues or inconsistencies. <figure class="wp-block-table"><table><tbody><tr><td>Given this Python…</td><td>Mypy will say…</td></tr><tr><td><code>x = 3</code></td><td>Success: no issues found in 1 source file</td></tr><tr><td><code>x: int = 3</code></td><td>Success: no issues found in 1 source file</td></tr><tr><td><code>x: str = 3</code></td><td>error: Incompatible types in assignment (expression has type “int”, variable has type “str”)  [assignment] Found 1 error in 1 file (checked 1 source file)</td></tr></tbody></table></figure> If something isn’t type annotated, mypy will try to infer the appropriate type. If you want to debug this, you can use the special function <code>reveal_type()</code>(only available in mypy). <figure class="wp-block-table resize-table"><table><tbody><tr><td>Given this Python…</td><td>Mypy will say…</td></tr><tr><td><code>x = 3 reveal_type(x)</code></td><td>note: Revealed type is “builtins.int” Success: no issues found in 1 source file</td></tr><tr><td><code>hex_digits = {     '0': 0, '1': 1, '2': 2, '3': 3,     '4': 4, '5': 5, '6': 6, '7': 7,     '8': 8, '9': 9, 'A': 10, 'B': 11,     'C': 12, 'D': 13, 'E': 14, 'F': 15,     'a': 10, 'b': 11, 'c': 12, 'd': 13,     'e': 14, 'f': 15 } reveal_type(hex_digits)</code> <code></code></td><td>note: Revealed type is “builtins.dict[builtins.str, builtins.int]”Success: no issues found in 1 source file </td></tr></tbody></table></figure> Most of the time, mypy is good at type inference, so it’s better to focus on annotating the parameters and return values of a function rather than the internal variables used in a function. If your codebase doesn’t already have a type checker, it’s also worth evaluating mypy’s competitors: <a href="https://google.github.io/pytype/">pytype</a>(Google), <a href="https://github.com/Microsoft/pyright">pyright</a> (Microsoft), and <a href="https://pyre-check.org/">pyre</a> (Meta). And for adding type annotations to existing Python code, take a look at <a href="https://monkeytype.readthedocs.io/en/stable/">MonkeyType</a> and <a href="https://github.com/JelleZijlstra/autotyping">autotyping</a>. <h2 class="wp-block-heading" id="h-an-example-of-detecting-downstream-issues">An example of detecting downstream issues</h2> <hr class="wp-block-separator has-text-color has-white-color has-alpha-channel-opacity has-white-background-color has-background is-style-wide"> Here’s a new function, <code>validate_cusip()</code>, that relies on the <code>cusip_checksum()</code> function from earlier: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">def cusip_checksum(cusip8: str) -> int: ... def validate_cusip(cusip: str) -> str | None: checksum: int if len(cusip) == 9: checksum = cusip_checksum(cusip[:8]) if str(checksum) == cusip[8]: return cusip else: return None elif len(cusip) == 8: checksum = cusip_checksum(cusip) return f"{cusip}{checksum}" else: return None</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>def cusip_checksum(cusip8: str) -> int: ... def validate_cusip(cusip: str) -> str | None: checksum: int if len(cusip) == 9: checksum = cusip_checksum(cusip[:8]) if str(checksum) == cusip[8]: return cusip else: return None elif len(cusip) == 8: checksum = cusip_checksum(cusip) return f"{cusip}{checksum}" else: return None</code></pre></div> Mypy is happy with this code: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">Success: no issues found in 1 source file</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>Success: no issues found in 1 source file</code></pre></div> Now, let’s say that we decide we should update <code>cusip_checksum()</code> to return <code>None</code> if it detects that the CUSIP is not valid: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">def cusip_checksum(cusip8: str) -> int | None: if len(cusip8) != 8: return None chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#" charmap: dict[str, int] = { char: value for value, char in enumerate(chars, start=0) } total: int = 0 for idx, char in enumerate(cusip8, start=0): try: value: int = charmap[char] except KeyError: return None if (idx % 2) == 1: value *= 2 total += (value // 10) + (value % 10) return (10 - total % 10) % 10</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>def cusip_checksum(cusip8: str) -> int | None: if len(cusip8) != 8: return None chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#" charmap: dict[str, int] = { char: value for value, char in enumerate(chars, start=0) } total: int = 0 for idx, char in enumerate(cusip8, start=0): try: value: int = charmap[char] except KeyError: return None if (idx % 2) == 1: value *= 2 total += (value // 10) + (value % 10) return (10 - total % 10) % 10</code></pre></div> Mypy automatically detects issues in how <code>validate_cusip()</code> is using <code>cusip_checksum()</code>: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">error: Incompatible types in assignment (expression has type "int | None", variable has type "int") [assignment]</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>error: Incompatible types in assignment (expression has type "int | None", variable has type "int") [assignment]</code></pre></div> Now that we’ve been alerted, we can update <code>validate_cusip()</code> to handle these changes: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<pre class="code-block-pro-copy-button-pre" aria-hidden="true"><textarea class="code-block-pro-copy-button-textarea" tabindex="-1" aria-hidden="true" readonly="">def cusip_checksum(cusip8: str) -> int | None: ... def validate_cusip(cusip: str) -> str | None: if len(cusip) == 9: match cusip_checksum(cusip[:8]): case int(checksum) if str(checksum) == cusip[8]: return cusip elif len(cusip) == 8: match cusip_checksum(cusip): case int(checksum): return f"{cusip}{checksum}" return None</textarea></pre><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>def cusip_checksum(cusip8: str) -> int | None: ... def validate_cusip(cusip: str) -> str | None: if len(cusip) == 9: match cusip_checksum(cusip[:8]): case int(checksum) if str(checksum) == cusip[8]: return cusip elif len(cusip) == 8: match cusip_checksum(cusip): case int(checksum): return f"{cusip}{checksum}" return None</code></pre></div> In this example, the functions were next to each other in the source code. But mypy really shines when the functions are spread across many files in the codebase. <h2 class="wp-block-heading" id="h-some-tips-and-tricks-for-using-type-annotations-effectively-nbsp">Some tips and tricks for using type annotations effectively </h2> <hr class="wp-block-separator has-text-color has-white-color has-alpha-channel-opacity has-white-background-color has-background is-style-wide"> Here are some specifics tips and tricks that I’ve learned through experience: <ol> <li> Understand the layers of your code. Focus on using type annotations in the inner layers before you focus on type annotations in the outer layers. </li> <li> Understand the limits of type annotations and when you need to add runtime validation. For example, when you are processing data from an external API, <code>typing.TypedDict</code> may be useful for type annotations, but it does not guarantee that the external API will actually provide that data – you will need to verify that the data matches your expected schema at runtime. </li> <li> Be careful about using primitive types (e.g. str, int) to represent domain ideas (e.g. securities, prices). Primitive types, like str/int/float, are easy to understand, but often lack the constraints needed for domain ideas. For example, you could represent a CUSIP like <code>my_cusip: str.</code> Even though a CUSIP can be represented as a string, not all strings are CUSIPs. A CUSIP is 9 characters, using an alphanumeric character set, and the 9th character is a checksum. For this example, you could increase the robustness of your code with a CUSIP type annotations, either by: <ul> <li class="alphabet"> Creating a CUSIP <a href="https://docs.python.org/3/library/dataclasses.html">dataclass</a> (or <a href="https://www.attrs.org/en/stable/">attrs</a> class)that takes a string and validates that the string is a valid CUSIP upon creation. </li> <li class="alphabet"> Using typing.NewType to denote that a CUSIP is a special type of string. For example, <code>CusipStr = NewType(“CusipStr”, str)</code>, and a <code>typing.TypeGuard</code> function like <code>is_cusip(text: str) -> TypeGuard[CusipStr]</code>. </li> </ul> </li> <li> typing.NewType and typing.Literal are lightweight tools for increasing type-safety.In (3.b), <code>CusipStr</code> is lightweight because at runtime the type is still str, without the extra baggage of <code>dataclass/attrs</code> classes. And <code>Literal</code> can act as a lightweight enum.Enum. For example, <ul> <li class="alphabet"> <code>order_status: Literal["open", "cancelled", "filled"]</code> </li> <li class="alphabet"> <code>order_status = "open".</code> <ul class="third"> <li>Mypy says: Success: no issues found in 1 source file.</li> </ul> </li> <li class="alphabet"> <code>order_status = "canceled"</code>. (Note the typo.) <ul class="third"> <li>Mypy says:<code>error: Incompatible types in assignment (expression has type "Literal['canceled']", variable has type "Literal['open', 'cancelled', 'filled']") [assignment].</code></li> </ul> </li> </ul> </li> </ol> Type annotations have substantial benefits for making your codebase more robust. They are not an all-or-nothing proposition — you can focus on adding type annotations to small parts of your codebase and growing the amount of type annotated code over time. Along with other technologies, Python’s type annotations help HRT to continue thriving in the fast-paced world of global trading.

2024/2/16

How Our Engineers Hot-Patched a Third Party Binary Library

At HRT, <a href="https://www.youtube.com/watch?v=ahOYPoDUHQ8" target="_blank" rel="noreferrer noopener">low-level engineers</a> face and solve as a team all kinds of technical challenges. This post documents a quite involved but particularly fun one. We had to delve into the intricacies of the system and learned a lot on the way. We hope you will too. <h2 class="wp-block-heading" id="h-introduction">Introduction</h2> <hr class="wp-block-separator has-text-color has-white-color has-alpha-channel-opacity has-white-background-color has-background"> Some time ago, HRT needed a production bug fixed in a third party binary-only <a href="https://en.wikipedia.org/wiki/X86-64" target="_blank" rel="noreferrer noopener">shared library</a>. The issue was quite pressing. However it can take a lot of time to put together a reproducer (if at all possible), send it to a vendor, and get an updated version. Luckily, we quickly discovered that the newest version of this library fixed the issue we were experiencing. This was going to be an easy fix, or so we thought! Throughout this post, we’ll call our current version of the library libfoo.so.1.1 and the new one libfoo.so.1.2. Note that we run <a href="http://www.kernel.org/" target="_blank" rel="noreferrer noopener">Linux</a> on <a href="https://en.wikipedia.org/wiki/X86-64">64-bit x86</a> computers. <h3 class="wp-block-heading" id="h-a-primer-on-shared-library-versioning">A primer on shared library versioning</h3> <hr class="wp-block-separator has-text-color has-white-color has-alpha-channel-opacity has-white-background-color has-background"> Shared libraries are usually linked to an application at runtime. However, if an application was built with one version of the library but an incompatible one is provided at runtime, the software will exhibit undefined behavior sooner or later. To detect this kind of issue, shared libraries come with a version string called a <a href="https://en.wikipedia.org/wiki/Soname" target="_blank" rel="noreferrer noopener">soname</a>.<a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footNotes">1</a> A change in soname simply means that the library is not backwards compatible. In other words, the application binary interface (ABI) of the build-time and runtime libraries are not compatible. At build time, applications are linked against the soname of the library. During a library installation, <a href="https://man7.org/linux/man-pages/man8/ldconfig.8.html" target="_blank" rel="noreferrer noopener">ldconfig</a> creates a symlink from the actual library to its soname. Let’s look at an example: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python /lib/x86_64-linux-gnu/libc.so.6 (0x00007fce0df8ce00) $ ls -l /lib/x86_64-linux-gnu/libc.so.6 lrwxrwxrwx 1 root root 12 Feb 20 2023 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.31.so* $ readelf -d /lib/x86_64-linux-gnu/libc-2.28.so | grep SONAME 0x000000000000000e (SONAME) Library soname: [libc.so.6]" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ ldd /bin/ls | grep libc libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fce0df8ce00) $ ls -l /lib/x86_64-linux-gnu/libc.so.6 lrwxrwxrwx 1 root root 12 Feb 20 2023 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.31.so* $ readelf -d /lib/x86_64-linux-gnu/libc-2.28.so | grep SONAME 0x000000000000000e (SONAME) Library soname: [libc.so.6]</code></pre></div> As you can see, ls is linked against a libc.so.6 symlink which points to the libc-2.31.so file (corresponding to <a href="https://www.gnu.org/software/libc/" target="_blank" rel="noreferrer noopener">glibc</a> 2.31 on this machine). Our distribution ships only libc-2.31.so in its glibc package. When the package is installed, ldconfig runs and creates the libc.6.so symlink. If we later install, say, glibc 2.32 that ships with the same soname (libc.so.6), nothing needs to be done after pointing the libc.6.so symlink to libc-2.32.so. The soname is the same, which means the library author has promised us that glibc 2.32 is backward compatible with 2.31. This is a very simple scheme that relies on a string in the binary and symlinks. No runtime check is actually done, so it’s quite important for libraries to have the correct soname. If there is a mismatch, software will crash in the most head-scratching way. <h4 class="wp-block-heading" id="h-back-to-our-library">Back to our library</h4> Surprisingly when we installed libfoo.so.1.2, our software would not start; the new library had a different soname. If we wanted to do things properly, this meant we had to rebuild all the involved software and potentially build two different versions of the library during the testing phase. This would add a significant amount of work and time to build, test, and deploy the fix.  Oddly enough, the new soname contained no version number. The previous soname was libfoo.so.1 and the new one was libfoo.so. It was quite strange to remove the versioning numbering; typically a new soname would have been libfoo.so.2. Unexpectedly, we also noticed that the new version was actually backwards compatible, so no change of soname was even necessary! We could have renamed the file from libfoo.so.1.2 to libfoo.so.1 to work around the soname change, but that would have prevented building against this library as linked software would be looking for libfoo.so. All of this would have resulted in a pretty janky setup. Our setup would be greatly simplified if we could modify the soname of the new library to match the previous one. We reached out to our vendor, but we couldn’t expect the kind of fast turnaround we’d like. As a result, we decided we would try to modify the soname of the binary to be libfoo.so.1 as it should be. <h4 class="wp-block-heading" id="h-attempting-a-quick-and-dirty-binary-edit">Attempting a quick-and-dirty binary edit</h4> As mentioned earlier, the soname is simply a string inside the binary. Could it be possible to just add the missing “.1” to our new library soname? Maybe it was followed by some padding bytes that we could overwrite? A programmer can dream! First we have to understand a little about the binary format used by shared libraries. Shared libraries on Linux are binary <a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format" target="_blank" rel="noreferrer noopener">ELF</a> files. This is quite a complex <a href="https://man7.org/linux/man-pages/man5/elf.5.html" target="_blank" rel="noreferrer noopener">format</a>. It’s divided into segments that contain different <a href="https://refspecs.linuxbase.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/sections.html" target="_blank" rel="noreferrer noopener">sections</a> that refer to each other through offsets: <figure class="wp-block-image aligncenter size-large is-resized"><img loading="lazy" decoding="async" width="574" height="1024" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/06/image2-jpg-574x1024.webp" alt="" class="wp-image-275836" style="width:400px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/06/image2-jpg-574x1024.webp 574w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/06/image2-jpg-480x856.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 574px, 100vw" referrerpolicy="no-referrer"></figure> Each section of the file may be or may not be mapped into memory at runtime. Sections that follow each other in the file may be mapped at different places in memory at runtime. There are all kinds of headers and offsets to keep track of that.  <figure class="wp-block-image aligncenter size-large is-resized"><img loading="lazy" decoding="async" width="1024" height="844" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/06/image1-jpg-1024x844.webp" alt="" class="wp-image-275837" style="width:700px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/06/image1-jpg-980x808.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/06/image1-jpg-480x396.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> The first order of business was to figure out where in the binary the soname is stored. If you’re lucky (and we were here), the string that contains the soname will be present only once, so finding its location is relatively easy. First, you need to run print the <a href="https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-73709.html" target="_blank" rel="noreferrer noopener">string table section</a> for dynamic linking (called .dynstr) with readelf -s .dynstr file and grep for the soname. This will show you the offset of the soname string in the .dynstr section. Then find the start of this section and add the offset to it to find the location of the same in the file.<a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footNotes">2</a> <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ readelf -p .dynstr libfoo.so.1.2 | grep libfoo.so [ 17ac] libfoo.so $ readelf -S libfoo.so.1.2 | grep dynstr [ 4] .dynstr STRTAB 0000000000003f00 00003f00 $ printf "0x%xn" $((0x17ac + 0x3f00)) 0x56ac $ xxd -s 0x56ac libfoo.so.1.2 | head -2 000056ac: 6c69 6266 6f6f 2e73 6f00 474c 4942 435f libfoo.so.GLIBC_ 000056bc: 322e 3300 474c 4942 435f 322e 3134 0047 2.3.GLIBC_2.14.G" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ readelf -p .dynstr libfoo.so.1.2 | grep libfoo.so [ 17ac] libfoo.so $ readelf -S libfoo.so.1.2 | grep dynstr [ 4] .dynstr STRTAB 0000000000003f00 00003f00 $ printf "0x%xn" $((0x17ac + 0x3f00)) 0x56ac $ xxd -s 0x56ac libfoo.so.1.2 | head -2 000056ac: 6c69 6266 6f6f 2e73 6f00 474c 4942 435f libfoo.so.GLIBC_ 000056bc: 322e 3300 474c 4942 435f 322e 3134 0047 2.3.GLIBC_2.14.G</code></pre></div> We could see that our “libfoo.so” soname is tightly packed with other strings. It already felt that our hack was likely to fail but we might as well try. We simply replaced the bytes 0x56b5 -> 0x56b7 with our string: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ xxd -s 0x56ac libfoo.so.1.2 | head -2 000056ac: 6c69 6266 6f6f 2e73 6f2e 3100 4942 435f libfoo.so.1.IBC_ 000056bc: 322e 3300 474c 4942 435f 322e 3134 0047 2.3.GLIBC_2.14.G" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ xxd -s 0x56ac libfoo.so.1.2 | head -2 000056ac: 6c69 6266 6f6f 2e73 6f2e 3100 4942 435f libfoo.so.1.IBC_ 000056bc: 322e 3300 474c 4942 435f 322e 3134 0047 2.3.GLIBC_2.14.G</code></pre></div> You might wonder why we were replacing existing bytes instead of inserting. The reason is that <a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format" target="_blank" rel="noreferrer noopener">ELF</a> is a binary format that refers to other sections (such as pointing to specific strings like the soname) by offsets. This means that if we were to insert anything, we’d have to find and rewrite many of these offsets. This is not something that can be done manually. This first attempt seemed to succeed. We could see the modified soname. When we tried to run ldconfig, it created the proper soname symlink. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ readelf -d libfoo.so.1.2 | grep SONAME 0x000000000000000e (SONAME) Library soname: [libfoo.so.1]" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ readelf -d libfoo.so.1.2 | grep SONAME 0x000000000000000e (SONAME) Library soname: [libfoo.so.1]</code></pre></div> However when we tried to run our software with this library, it would not start due to a missing symbol! Let’s dump the <a href="https://refspecs.linuxbase.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/symversion.html#SYMVERRQMTS" target="_blank" rel="noreferrer noopener">version need information</a> of the patched and unpatched libraries: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">ShellScript<span role="button" tabindex="0" data-code="$ ./oursoftware ./oursoftware: /lib64/ld-linux-x86-64.so.2: version `1' not found (required by libfoo.so.1) $ readelf -V libfoo.so.1.2 | grep -A3 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: ld-linux-x86-64.so.2 Cnt: 1 0x0010: Name: 1 Flags: none Version: 5 $ readelf -V libfoo.so.1.2.old | grep -A3 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: ld-linux-x86-64.so.2 Cnt: 1 0x0010: Name: GLIBC_2.3 Flags: none Version: 5" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ ./oursoftware ./oursoftware: /lib64/ld-linux-x86-64.so.2: version `1' not found (required by libfoo.so.1) $ readelf -V libfoo.so.1.2 | grep -A3 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: ld-linux-x86-64.so.2 Cnt: 1 0x0010: Name: 1 Flags: none Version: 5 $ readelf -V libfoo.so.1.2.old | grep -A3 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: ld-linux-x86-64.so.2 Cnt: 1 0x0010: Name: GLIBC_2.3 Flags: none Version: 5</code></pre></div> The unpatched library had a dependency on a GLIBC_2.3 symbol that became a 1 in our patched file. As noted above, the ELF format uses offset to refer to things. And it appears that the bytes we replaced are still referred to. We had replaced GL with 1\x0 so the linker thought that it needed to find a 1 symbol in ld-linux-x86-64.so.2 instead of GLIBC_2.3. This was getting complicated! Surely there was a tool that we could use. <h4 class="wp-block-heading" id="h-trying-patchelf">Trying patchelf</h4> <a href="https://github.com/NixOS/patchelf" target="_blank" rel="noreferrer noopener">Patchelf</a> is a nifty tool that allows you to modify binary <a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format" target="_blank" rel="noreferrer noopener">ELF</a> files (such as shared libraries). In particular, it can also change the soname of a library. We ran patchelf on our shared library passing the old library’s soname. The readelf command above confirmed it now had the previous soname. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ readelf -d libfoo.so.1.2 | grep -i soname 0x000000000000000e (SONAME) Library soname: [libfoo.so] $ patchelf --set-soname libfoo.so.1 libfoo.so.1.2 $ readelf -d libfoo.so.1.2 | grep -i soname 0x000000000000000e (SONAME) Library soname: [libfoo.so.1]" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ readelf -d libfoo.so.1.2 | grep -i soname 0x000000000000000e (SONAME) Library soname: [libfoo.so] $ patchelf --set-soname libfoo.so.1 libfoo.so.1.2 $ readelf -d libfoo.so.1.2 | grep -i soname 0x000000000000000e (SONAME) Library soname: [libfoo.so.1]</code></pre></div> We high fived and confidently started to test the new library, almost certain that we were done. We fired up <a href="https://man7.org/linux/man-pages/man1/ldd.1.html" target="_blank" rel="noreferrer noopener">ldd</a> to see if our library was getting picked up <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ ldd ./oursoftware ./oursoftware: error while loading shared libraries: libfoo.so.1: ELF load command address/offset not properly aligned" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ ldd ./oursoftware ./oursoftware: error while loading shared libraries: libfoo.so.1: ELF load command address/offset not properly aligned</code></pre></div> Let’s see what patchelf did to our software. First we noticed that it appended the new soname to the .dynstr section (instead of modifying it) and moved that section at the end of the binary. This is loaded with a new <a href="https://refspecs.linuxbase.org/elf/gabi4+/ch5.pheader.html" target="_blank" rel="noreferrer noopener">program header</a> <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ readelf -p .dynstr libfoo.so.1 | grep libfoo.so [ 194c] libfoo.so [ 1a0c] libfoo.so.1 $ readelf -S libfoo.so.1 | grep dynstr [29] .dynstr STRTAB 0000000000402ac8 0004aac8 $ readelf -l libfoo.so.1 (...) Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align (...) LOAD 0x000000000004a000 0x0000000000400000 0x0000000000400000 0x0000000000002588 0x0000000000002588 RW 0x200000" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ readelf -p .dynstr libfoo.so.1 | grep libfoo.so [ 194c] libfoo.so [ 1a0c] libfoo.so.1 $ readelf -S libfoo.so.1 | grep dynstr [29] .dynstr STRTAB 0000000000402ac8 0004aac8 $ readelf -l libfoo.so.1 (...) Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align (...) LOAD 0x000000000004a000 0x0000000000400000 0x0000000000400000 0x0000000000002588 0x0000000000002588 RW 0x200000</code></pre></div> The 3rd party library had its LOAD segments configured to be loaded on a 2MiB boundary (0x200000) and patchelf did not handle that gracefully. Since this didn’t matter to us here, we simply modified patchelf to forcibly change the alignment back to a regular page size. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++::rewriteSectionsLibrary() wri(phdr.p_vaddr, wri(phdr.p_paddr, startPage)); wri(phdr.p_filesz, wri(phdr.p_memsz, neededSpace)); wri(phdr.p_flags, PF_R | PF_W); - wri(phdr.p_align, alignStartPage); + wri(phdr.p_align, getPageSize()); } normalizeNoteSegments();" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>--- a/src/patchelf.cc +++ b/src/patchelf.cc @@ -897,7 +897,7 @@ void ElfFile<ElfFileParamNames>::rewriteSectionsLibrary() wri(phdr.p_vaddr, wri(phdr.p_paddr, startPage)); wri(phdr.p_filesz, wri(phdr.p_memsz, neededSpace)); wri(phdr.p_flags, PF_R | PF_W); - wri(phdr.p_align, alignStartPage); + wri(phdr.p_align, getPageSize()); } normalizeNoteSegments();</code></pre></div> Are we done? Our hopes are quickly dashed. When we ran ldconfig, it failed by printing “file libfoo.so.1.2 is truncated.” 🫤 ldconfig is failing while trying to load the dynamic string section that patchelf just added. Let’s see what it <a href="https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=elf/readelflib.c;hb=5a82c74822d3272df2f5929133680478c0cfb4bd" target="_blank" rel="noreferrer noopener">expects</a>: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++d_tag != DT_NULL; ++dyn_entry) { check_ptr (dyn_entry); if (dyn_entry->d_tag == DT_STRTAB) { /* ... */ dynamic_strings = (char *) (file_contents + dyn_entry->d_un.d_val- loadaddr);" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>/* Find the string table. */ dynamic_strings = NULL; for (dyn_entry = dynamic_segment; dyn_entry->d_tag != DT_NULL; ++dyn_entry) { check_ptr (dyn_entry); if (dyn_entry->d_tag == DT_STRTAB) { /* ... */ dynamic_strings = (char *) (file_contents + dyn_entry->d_un.d_val- loadaddr);</code></pre></div> The code is trying to load this section by treating the d_val field as an offset where the file is mapped. The dyn_entry type is: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<span role="button" tabindex="0" data-code="typedef struct { unsigned char d_tag[8]; /* entry tag value */ union { unsigned char d_val[8]; unsigned char d_ptr[8]; } d_un; } Elf64_External_Dyn;" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>typedef struct { unsigned char d_tag[8]; /* entry tag value */ union { unsigned char d_val[8]; unsigned char d_ptr[8]; } d_un; } Elf64_External_Dyn;</code></pre></div> Now we looked at how patchelf <a href="https://github.com/NixOS/patchelf/blob/afd3cc94450dc6845e923f9221fa38818538219e/src/patchelf.cc#L1380" target="_blank" rel="noreferrer noopener">modifies</a> the soname. The approach is quite straightforward: create a section that can accommodate the new soname, simply add the new soname at the end of the section and append the new segment at the end of the file. When patchelf <a href="https://github.com/NixOS/patchelf/blob/afd3cc94450dc6845e923f9221fa38818538219e/src/patchelf.cc#L1192" target="_blank" rel="noreferrer noopener">writes</a> the new section and populates the Elf64_External_Dyn struct described above, it does: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++d_tag)) != DT_NULL; dyn++) if (d_tag == DT_STRTAB) dyn->d_un.d_ptr = findSectionHeader(".dynstr").sh_addr;" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code> for (auto dyn = dyn_table; (d_tag = rdi(dyn->d_tag)) != DT_NULL; dyn++) if (d_tag == DT_STRTAB) dyn->d_un.d_ptr = findSectionHeader(".dynstr").sh_addr;</code></pre></div> The first thing we noticed was that they use different fields — d_val in one, d_ptr in the other. Though these are the same type and size and part of the same union, the field is treated as an offset in ldconfig but as an address in patchelf. Let’s change patchelf again to try to work around this issue: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++::rewriteHeaders(Elf_Addr phdrAddress) unsigned int d_tag; for (auto dyn = dyn_table; (d_tag = rdi(dyn->d_tag)) != DT_NULL; dyn++) if (d_tag == DT_STRTAB) - dyn->d_un.d_ptr = findSectionHeader(".dynstr").sh_addr; + dyn->d_un.d_val = findSectionHeader(".dynstr").sh_offset; else if (d_tag == DT_STRSZ) dyn->d_un.d_val = findSectionHeader(".dynstr").sh_size; else if (d_tag == DT_SYMTAB)" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>@@ -1191,7 +1191,7 @@ void ElfFile<ElfFileParamNames>::rewriteHeaders(Elf_Addr phdrAddress) unsigned int d_tag; for (auto dyn = dyn_table; (d_tag = rdi(dyn->d_tag)) != DT_NULL; dyn++) if (d_tag == DT_STRTAB) - dyn->d_un.d_ptr = findSectionHeader(".dynstr").sh_addr; + dyn->d_un.d_val = findSectionHeader(".dynstr").sh_offset; else if (d_tag == DT_STRSZ) dyn->d_un.d_val = findSectionHeader(".dynstr").sh_size; else if (d_tag == DT_SYMTAB)</code></pre></div> Hooray, ldconfig now parsed the file properly and created the libfoo.so.1 symlink pointing to libfoo.so.1.2. Unfortunately, when we tried to start our software, it crashed right away. 😭 It turns out that the string mapped at the offset is present in memory but actually <a href="https://man7.org/linux/man-pages/man2/mprotect.2.html" target="_blank" rel="noreferrer noopener">mprotected</a> with PROT_NONE by the <a href="https://sourceware.org/git/?p=glibc.git;a=blob;f=elf/dl-map-segments.h;h=ac10182d587ad00229fc1c89c4c5aa4ace4a9251;hb=HEAD" target="_blank" rel="noreferrer noopener">runtime loader</a> because our patched ELF file now has non-contiguous PT_LOAD segments. 🤯 Note that the runtime loader would actually be happy with the original patchelf code where sh_addr is used. This meant that the only way for both the loader and ldconfig<a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footNotes">3</a> (and potentially other parts of the system) to properly parse and use our patched ELF file was to make a large number of changes to patchelf. There had to be an easier way. <h4 class="wp-block-heading" id="h-back-to-editing-the-file-manually">Back to editing the file manually</h4> Our initial approach almost worked except for this one version requirement that was pointing to a string we wanted to change. It occurred to us that if we could just drop it, things would work just fine. This was a version requirement for glibc 2.3 which is long gone.<a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footNotes">4</a> If you’re going to ship a library to hundreds of customers, removing dependencies is a bad idea. But in our limited environment where we knew this requirement was fulfilled, this dependency was simply useless. Simply dropping the entry would cause many offsets to change and create even more issues. However, we could simply override it to point to something else. Let’s look again at the requirements: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ readelf -V libfoo.so.1.2.old | grep -A5 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: ld-linux-x86-64.so.2 Cnt: 1 0x0010: Name: GLIBC_2.3 Flags: none Version: 5 0x0020: Version: 1 File: libc.so.6 Cnt: 6 0x0030: Name: GLIBC_2.14 Flags: none Version: 8" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ readelf -V libfoo.so.1.2.old | grep -A5 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: ld-linux-x86-64.so.2 Cnt: 1 0x0010: Name: GLIBC_2.3 Flags: none Version: 5 0x0020: Version: 1 File: libc.so.6 Cnt: 6 0x0030: Name: GLIBC_2.14 Flags: none Version: 8</code></pre></div> We would like to avoid pointing to the GLIBC_2.3 string. What if we simply duplicated the 2nd requirement over the 1st one? Nothing would be pointing to the truncated GLIBC_2.3. Let’s look at the <a href="https://refspecs.linuxfoundation.org/LSB_1.3.0/gLSB/gLSB/symverrqmts.html" target="_blank" rel="noreferrer noopener">version requirement structures</a>: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<span role="button" tabindex="0" data-code="typedef struct { Elf64_Half vn_version; Elf64_Half vn_cnt; Elf64_Word vn_file; Elf64_Word vn_aux; Elf64_Word vn_next; } Elf64_Verneed; typedef struct { Elf64_Word vna_hash; Elf64_Half vna_flags; Elf64_Half vna_other; Elf64_Word vna_name; Elf64_Word vna_next; } Elf64_Vernaux;" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>typedef struct { Elf64_Half vn_version; Elf64_Half vn_cnt; Elf64_Word vn_file; Elf64_Word vn_aux; Elf64_Word vn_next; } Elf64_Verneed; typedef struct { Elf64_Word vna_hash; Elf64_Half vna_flags; Elf64_Half vna_other; Elf64_Word vna_name; Elf64_Word vna_next; } Elf64_Vernaux;</code></pre></div> Elf64_Verneed describes the lines printed as File above and is followed by one or more Elf64_Vernaux representing the lines with Names above. Now let’s print this in hex: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ xxd -s 0x59e8 libfoo.so.1.2 | head -4 000059e8: 0100 0100 2419 0000 1000 0000 2000 0000 ....$....... ... 000059f8: 1369 690d 0000 0500 5c19 0000 0000 0000 .ii............ 00005a08: 0100 0600 0f19 0000 1000 0000 0000 0000 ................ 00005a18: 9491 9606 0000 0800 6619 0000 1000 0000 ........f......." style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ xxd -s 0x59e8 libfoo.so.1.2 | head -4 000059e8: 0100 0100 2419 0000 1000 0000 2000 0000 ....$....... ... 000059f8: 1369 690d 0000 0500 5c19 0000 0000 0000 .ii............ 00005a08: 0100 0600 0f19 0000 1000 0000 0000 0000 ................ 00005a18: 9491 9606 0000 0800 6619 0000 1000 0000 ........f.......</code></pre></div> Line 3 is the Elf64_VerNeed we wanted to replace with line 3. Line 4 is the Elf64_Vernaux we wanted to replace with line 4. There was just one small adjustment we needed to make. The first file had only one Elf64_Vernaux entry while the second one had 6 (the Cnt field displayed by readelf above). When copying line 3, we simply changed the third byte from 0x6 to 0x1. We’re simply changing the least significant byte in vn_cnt. After this change our requirements look like: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(14px, .875rem, 21px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:clamp(20px, 1.25rem, 30px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">Python<span role="button" tabindex="0" data-code="$ readelf -V libfoo.so.1.2 | grep -A5 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: libc.so.6 Cnt: 1 0x0010: Name: GLIBC_2.14 Flags: none Version: 8 0x0020: Version: 1 File: libc.so.6 Cnt: 6 0x0030: Name: GLIBC_2.14 Flags: none Version: 8" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>$ readelf -V libfoo.so.1.2 | grep -A5 "Version needs" Version needs section '.gnu.version_r' contains 2 entries: Addr: 0x00000000000059e8 Offset: 0x0059e8 Link: 4 (.dynstr) 000000: Version: 1 File: libc.so.6 Cnt: 1 0x0010: Name: GLIBC_2.14 Flags: none Version: 8 0x0020: Version: 1 File: libc.so.6 Cnt: 6 0x0030: Name: GLIBC_2.14 Flags: none Version: 8</code></pre></div> The output looks quite silly with the duplicated requirement. But with this last set of changes, the library could be loaded flawlessly by all the tools on the system. We could finally declare victory! 🥳 <h4 class="wp-block-heading" id="h-conclusion">Conclusion</h4> This was certainly a journey. It was quite satisfying for our team to have managed to solve this issue. Our vendor eventually reached out and agreed to fix the soname in their next revision so all is well! If you have found this post interesting, check out other engineering-oriented <a href="https://www.hudsonrivertrading.com/hrtbeat/category/engineering/" target="_blank" rel="noreferrer noopener">posts</a> on our <a href="https://www.hudsonrivertrading.com/hrtbeat/" target="_blank" rel="noreferrer noopener">blog</a>. If you would like to know more about HRT, check out our <a href="https://www.hudsonrivertrading.com/" target="_blank" rel="noreferrer noopener">website</a> or consider applying to <a href="https://www.hudsonrivertrading.com/careers/" target="_blank" rel="noreferrer noopener">join our team</a>! <h4 class="wp-block-heading" id="h-further-reading">Further reading</h4> The <a href="https://www.johnlevine.com/books.phtml" target="_blank" rel="noreferrer noopener">Linkers and loaders</a> book by John R. Levine The <a href="https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-AMD64/LSB-Core-AMD64/book1.html" target="_blank" rel="noreferrer noopener">Linux Standards Base</a> for x86_64 The ELF format <a href="https://github.com/corkami/pics/blob/master/binary/elf101/elf101.pdf" target="_blank" rel="noreferrer noopener">poster</a> <h4 class="wp-block-heading" id="h-appendix-binary-compatibility-in-shared-libraries">Appendix: Binary compatibility in shared libraries</h4> As we mentioned in our post, the soname of a library must be changed if it is not backwards compatible. Backwards compatibility for libraries is not always well understood so we thought we’d give some background about it. Backwards compatibility means that code compiled and linked with an older version of the library will run just the same if the shared library is upgraded without recompiling the dependent code. It’s a crucial property for shared libraries as upgrading both libraries and dependent code in lockstep is a lot more involved than just copying in a new shared library. It makes potentially reverting a lot more difficult. If a shared library is backwards compatible, users can upgrade the libraries across their system then start recompiling the code against it. Each step is safe and should be easy to revert. Ensuring backward compatibility is not trivial and requires understanding of the platform <a href="https://gitlab.com/x86-psABIs/x86-64-ABI/">ABI</a>. However these are some simple things that can be done to maintain backwards compatibility: <ul class="wp-block-list"> <li>Adding a new function or a non-virtual method to a library</li> <li>Adding new types</li> <li>Any non-inlined logic changes (for example adding a check in an existing function that some pointer is not NULL) that does not change the contract of the library.</li> <li>Adding new static or global variables</li> </ul> A very common thing that we’d like to discuss is adding new fields to existing types used in the public interface. It’s not as straightforward as other changes we mentioned above but it’s possible to do it with a little bit of planning. The basic idea is to only add fields at the end of a struct and always pass these by address and give the new code a way to distinguish between the older and the newer structures at runtime. To do that, there are two very straightforward ways: <ol class="wp-block-list"> <li>Passing the size of the struct along with the pointer. It’s tedious but it’s bullet proof. The size argument can be automatically added by the use of forced inline function.</li> <li>Adding a flag field that indicates which fields are present in the struct. This technique can also be hidden from the user by using forced inline constructors.</li> </ol> This is just barely an introduction on the subject. For a more in-depth discussion on this topic and more advanced techniques, we recommend <a href="https://akkadia.org/drepper/dsohowto.pdf" target="_blank" rel="noreferrer noopener">How to write shared libraries</a> by Ulrich Drepper and <a href="https://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-library-handles-backward-compatibility#" target="_blank" rel="noreferrer noopener">How the GNU C library handles backwards compatibility</a>. <hr class="wp-block-separator has-alpha-channel-opacity"> 1.“readelf -d file | grep SONAME” can be used to display the soname in “file”. The readelf tool is part of the <a href="https://www.gnu.org/software/binutils/">binutils</a> package.<a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footnote1">↩︎</a> 2. We use xxd which ships with the vim editor to display a hex dump but any tool to dump hexadecimal would do. <a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footnote2">↩︎</a> 3. This turned out to be a bug in readelflib.c in glibc which was <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=25087">fixed</a> in glibc 2.31 <a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footnote3">↩︎</a> 4. It was released in 2002: <a href="https://sourceware.org/glibc/wiki/Glibc%20Timeline">https://sourceware.org/glibc/wiki/Glibc%20Timeline</a> <a href="https://www.hudsonrivertrading.com/hrtbeat/how-our-engineers-hot-patched-a-third-party-binary-library/#footnote4">↩︎</a>

2023/12/7

Intern Spotlight: 2023 Software Engineering Summer Projects

<h3 class="wp-block-heading" id="h-introduction">Introduction</h3> Throughout the summer, HRT’s Software Engineering interns worked with mentors, teams, and longtime HRTers on independent programming projects that tackled the most interesting problems in the world of automated trading, making a notable impact on our day-to-day operations. This post in our intern spotlight series highlights the work of three of our Software Engineering interns: <ul class="wp-block-list"> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-software-engineering/#h-generating-code-with-code-reflection">Generating Code with Code Reflection by Jacob Urbanczyk</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-software-engineering/#h-heartbeat-v2">Heartbeat v2 by Alice Chen</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-software-engineering/#h-unraveling-exception-backtraces">Unraveling Exception Backtraces by Isidor Kaplan </a></li> </ul> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-generating-code-with-code-reflection">Generating Code with Code Reflection</h2> <h3 class="wp-block-heading" id="h-by-jacob-urbanczyk">by Jacob Urbanczyk</h3> <div style="height:25px" aria-hidden="true" class="wp-block-spacer"></div> <h4 class="wp-block-heading" id="h-intro">Intro</h4> Trading on dozens of exchanges is challenging. Many of them use their own communication protocol formats. One laborious problem HRT faces is writing and maintaining the code interfacing exchanges. It is crucial to abstract out the common parts to avoid repetition and reduce maintenance toil.  HRT records every message it sends to an exchange for various purposes. Messages typically have an exchange-specific encoding, but often share a similar structure: a message header and payload. The header consists of information like the total message length and type. <figure class="wp-block-image aligncenter size-full is-style-default full-width-img"><img loading="lazy" decoding="async" width="624" height="84" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2.png" alt="" class="wp-image-273594" style="object-fit:contain" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2.png 624w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2-480x65.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 624px, 100vw" referrerpolicy="no-referrer"></figure> Messages are consumed and replayed by many of our verification tools. These tools use the TripRaw library (Trip is our internal system for sending orders), which reads and dispatches messages differently based on each message and protocol type for internal processing. Since we don’t know the message type beforehand, we must first parse a few bytes, read the type, and then proceed with the parsing procedure.  This repetitive code could be avoided if we could generically infer the message type. My project involved leveraging RawReflect, our internal C++ reflection library, to find the fullest type and get rid of the repetitive part of TripRaw. <h4 class="wp-block-heading" id="h-rawreflect">RawReflect</h4> Reflection is a program’s ability to examine and introspect its own structure and code at run-time. RawReflect uses the program’s types to perform bidirectional conversion between a binary message’s representations and a structured object. To illustrate this, consider the following C++ struct: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>struct Foobar { int field1; short field2; char str[3]; };</code></pre></div> RawReflect is able to convert a bag-of-bytes representing this struct into a structured message representation: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++ Foobar(field1=1000, field2=66, str='HRT')" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>000003E80042485254 --> Foobar(field1=1000, field2=66, str='HRT')</code></pre></div> This structured message contains all field types, names, and offsets in the struct. It exposes a convenient interface for making queries like “what is the value of the field <code>str</code>?”  RawReflect supports nested structs, unions, variable length structs, and automatic upcasting. RawReflect determines the most concrete type by parsing the message header and then inferring the fullest type. Supporting RawReflect for new structs simply requires annotating the struct with a couple of preprocessor macros. The build system will do the rest by using clang to dump struct memory layouts and then auto-generating all necessary code. As part of my project, I worked on making RawReflect more robust. The base type that is being upcasted from can now be the first member in a struct, instead of inheriting from the base. Moreover, the trailing data of a struct itself can be now upcasted to the fullest type. This allows the trailing data to be an arbitrarily-sized batch of messages, each of a different type: <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="624" height="133" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-3.png" alt="" class="wp-image-273595" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-3.png 624w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-3-480x102.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 624px, 100vw" referrerpolicy="no-referrer"></figure> <h4 class="wp-block-heading" id="h-solution">Solution</h4> RawReflect solves the problem of finding the fullest type of a raw message in a generic way. TripRaw can take advantage of that to avoid the necessity of writing boilerplate code for each exchange protocol format. When adding support for a new protocol to TripRaw: <ol class="wp-block-list pad-left"> <li>Annotate protocol structs to make it reflectable</li> <li>Dump struct layouts using clang</li> <li>Generate RawReflect and TripRaw code stubs</li> </ol> Steps 2-3 are done automatically by the build system. The clients of TripRaw can now use messages from the newly added protocol! Reflection is a powerful programming paradigm that can reduce the amount of boilerplate code and thus the maintenance burden. The trade off here is the performance drop — this is fast enough for batch processing of recorded data, but not for latency-sensitive use. However, it is a perfect fit for TripRaw, and has potential for wider adoption across the firm. <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-heartbeat-v2">Heartbeat v2</h2> <h3 class="wp-block-heading" id="h-by-alice-chen">By Alice Chen</h3> <div style="height:25px" aria-hidden="true" class="wp-block-spacer"></div> <h4 class="wp-block-heading" id="h-the-problem">The Problem</h4> HRT’s build and test tools spawn many child processes within a large process tree, parts of which run on remote machines over SSH. If interrupts occur while running these processes, such as SIGINTs triggered by Ctrl+C’s, we expect all processes in the process tree to receive the signal. To gracefully handle these interrupts, build/autotest installs custom signal handlers that perform cleanup tasks like generating user-friendly diff summaries. For some processes, reliably invoking these signal handlers is necessary for correctness. If all processes are executing on the same machine, the operating system running on that host is responsible for ensuring that all processes in that process group receive the signal. However, in the case of remote processes, signal propagation over SSH tunnels is fragile, as the operating system is unable to guarantee that processes running on other hosts receive the signal properly. As a result, users might see old remote build processes lingering even after the forefront local build was interrupted, which hogs machine resources and causes unexpected side effects for developers. <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="617" height="406" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1.png" alt="" class="wp-image-273596" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1.png 617w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1-480x316.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 617px, 100vw" referrerpolicy="no-referrer"></figure> Figure 1: The difficulties of propagating signals across machine boundaries <h4 class="wp-block-heading" id="h-existing-work">Existing Work</h4> Heartbeat v.1 was our first attempt to properly propagate signals through a process tree. In this version, we propagated signals by logging signals intercepted by the root process into an NFS file, which are readable across host machines. Child processes then periodically read the file to keep up-to-date and properly invoke signal handlers. Unfortunately, NFS with caching has poor consistency guarantees and the filers could become overburdened, making it difficult to ensure that the data read is fresh. <h4 class="wp-block-heading" id="h-heartbeat-v-2-grpc-heartbeat-client-server">Heartbeat v.2: gRPC Heartbeat client-server</h4> My intern project, Heartbeat v.2, eliminates the reliance on NFS and turns to gRPC as the communication mechanism between the root and all of the clients in the process tree. At the startup of a new root process, we spawn a gRPC heartbeat server alongside the root process that acts as the central authority for intercepting and tracking signals. Heartbeat clients interact with the server through protobuf ping messages: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(2 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<span role="button" tabindex="0" data-code="(heartbeat.proto) message Heartbeat { double ts = 1; Process process = 2; Signals signals = 3; string log = 4; bool shutdown = 5; } message HeartbeatState { double ts = 1; Signals signals = 2; } (heart_server.cc) grpc::Status Ping(grpc::ServerContext* context, const heartbeat::Heartbeat* beat, heartbeat::HeartbeatState* state) override;" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>(heartbeat.proto) message Heartbeat { double ts = 1; Process process = 2; Signals signals = 3; string log = 4; bool shutdown = 5; } message HeartbeatState { double ts = 1; Signals signals = 2; } (heart_server.cc) grpc::Status Ping(grpc::ServerContext* context, const heartbeat::Heartbeat* beat, heartbeat::HeartbeatState* state) override;</code></pre></div> As seen in the HeartbeatState message above, the gRPC heartbeat server responds to ping messages from the clients with the current status of signals received. As a nice side effect, clients can now forward messages logged in their process to the root server, which aggregates all messages into a single, unified log. <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="624" height="403" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-2.png" alt="" class="wp-image-273597" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-2.png 624w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-2-480x310.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 624px, 100vw" referrerpolicy="no-referrer"></figure> Figure 2: Heartbeat v.2’s high level design <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="468" height="498" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05-1.png" alt="" class="wp-image-273598" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05-1.png 468w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05-1-282x300.png 282w" sizes="(max-width: 468px) 100vw, 468px" referrerpolicy="no-referrer"></figure> Figure 3: Control flow of Heartbeat v.2 As seen in figure 2 above, each Heartbeat v.2 child process starts its gRPC client in a separate heartbeat thread, which periodically pings the heartbeat server to obtain the most recent state of signals; if there are pending log messages, the client will piggyback the new log messages along with the ping. After receiving a response from the server, the heartbeat thread checks if any new signals have been received, and throws any new signals back to the main thread’s handlers. Integrating Heartbeat v.2 into HRT’s build and test tools is very non-intrusive and transparent to the user since it only requires exec-ing the server/clients alongside the build/test processes in the tree. The users are able to run the same commands as before, and automatically enable Heartbeat v.2 under the hood, giving them the benefit of better signal handling at essentially zero cost. <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-unraveling-exception-backtraces">Unraveling Exception Backtraces</h2> <h3 class="wp-block-heading" id="h-by-isidor-kaplan">by Isidor Kaplan</h3> <div style="height:25px" aria-hidden="true" class="wp-block-spacer"></div> <h4 class="wp-block-heading" id="h-background">Background</h4> C++’s exception handling is not traditionally celebrated for its user-friendly interface, particularly when compared to languages like Python. At Hudson River Trading, developers extensively use exceptions in non-performance-critical paths, and the minimal information available when an exception is thrown often costs us time in the debugger. An experience more like Python’s, where an uncaught exception shows the source locations of each frame it passed through, would make many more problems diagnosable at a glance. Recently, <a href="https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p2490r3.html">it was proposed for C++ standardization</a> that the catcher of an exception should be able to request that a stacktrace be included with it; the proposal included <a href="https://github.com/ecatmur/stacktrace-from-exception">sample code</a> showing how to achieve this on current C++ implementations. HRT recently adapted this logic for internal use, but uptake has been limited since our facilities for displaying the resulting backtraces were fairly primitive. This summer, I overhauled HRT’s backtrace library, significantly enhancing HRT’s workflow concerning exceptions. <h4 class="wp-block-heading" id="h-backtrace-improvements">Backtrace Improvements</h4> During my internship at HRT, I overhauled the internal backtrace framework to provide a modern interface to fetch binary names, symbol names, source code locations, and snippets from captured frame pointers. The updated product supports different levels of granularity and works with the internal exception class. As a result, developers have a better experience when debugging production issues. The backtrace library serves three main use cases:  <ol start="1" class="wp-block-list pad-left"> <li>Resolving and printing exception information upon request from C++ inside a catch clause (screenshot below) </li> <li>Providing full context from the std::terminate handler </li> <li>Propagating C++ backtraces into Python when the exception crosses the boundary (we extensively call into our C++ code via Python bindings)</li> </ol> <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="980" height="617" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-2.webp" alt="" class="wp-image-273600" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-2.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-2-480x302.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="980" height="609" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07-1.png" alt="" class="wp-image-273601" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07-1.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07-1-480x298.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> Balancing information granularity and performance overhead is vital in high-performance systems. The tool supports various detail levels, manually specified or implied based on environment considerations. At a minimum, it decodes an instruction pointer into a binary+offset pair so that we have enough information to resolve symbols later, even when using dynamic linking (mostly relevant for the Python-bindings case). In critical systems, this is the sole computation, deferring the more time-consuming parts for after-the-fact computation using an internal tool analogous to <a href="https://lwn.net/Articles/592724/">Linux’s decode_stacktrace</a>. In less latency-sensitive contexts, more data can be shown to reduce debugging work, which involves reading the binary’s debug symbol table to identify symbol+offset and function location. Taking a step further, we may locate the file:line in the source code. These operations have similar overhead and are tolerable outside critical paths. We may also include actual source code in the output to provide developers with more insight without external tools. However, loading large source files from network-attached storage can be slow for live execution processes, so we only print this information when the process will terminate or if the caller is from Python, where performance is less critical. Moreover, to allow Python users calling C++ to get meaningful exception messages, I modified <a href="https://github.com/wjakob/nanobind">nanobind</a> and <a href="https://github.com/pybind/pybind11">pybind11</a> to capture the C++ stacktrace using the previous primitive and attached it to the converted Python exception. The wrapped Python exception by default only includes frame pointers, and the symbol lookup is deferred until printing the exception. Other challenges involve optimizing performance through caching, handling network storage, and taking care of missing sections of <a href="https://en.wikipedia.org/wiki/DWARF">DWARF</a> or symbol table. With everything addressed properly, developers at HRT may now spend less time loading core dumps or guessing the source of C++ exceptions from Python messages.

2023/11/1

Intern Spotlight: 2023 HRT AI Labs Summer Projects

<h3 class="wp-block-heading">Introduction</h3> HAIL (HRT AI Labs) once again hosted many Algo Developer interns during the HRT 2023 Algo Summer Internship program. Like <a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-hrt-ai-labs-summer-projects/">last year</a>, these interns tackled a broad variety of problems that the HAIL team works on every day to apply artificial intelligence in trading. As one would expect from such an exceptional group, these interns made serious contributions to our research projects in the short five weeks they spent on a rotation with our team. This post will delve into three of those projects: This post in our intern spotlight series highlights the work of three of our Software Engineering interns: <ul class="wp-block-list"> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-hrt-ai-labs-summer-projects/#h-signal-architecture-search-with-diverse-multi-solution-genetic-algorithms">Signal Architecture Search with Diverse Multi-Solution Genetic Algorithms by Jesse Chan</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-hrt-ai-labs-summer-projects/#h-speed-up-hyperparameter-search-with-graybox-bayesian-optimization">Speed Up Hyperparameter Search with Graybox Bayesian Optimization by Ziqian Zhong</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2023-hrt-ai-labs-summer-projects/#h-accelerating-quantitative-finance-with-gpu-computation">Accelerating Quantitative Finance with GPU Computation by Emma Yang</a></li> </ul> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-signal-architecture-search-with-diverse-multi-solution-genetic-algorithms">Signal Architecture Search with Diverse Multi-Solution Genetic Algorithms</h2> <h3 class="wp-block-heading">By Jesse Chan</h3> A key part of algorithmic trading is the ability to anticipate price movements. Finding predictive “signals” is a difficult task; traditionally, signals were hand-crafted by researchers by blending market intuition with empirical analysis of market data. Although market intuition is helpful for signal creation, the space of possible signals is very large. It is reasonable to ask if we could utilize AI to craft signals that outperform or are complementary to the signals that humans create. In this project, we used tick-level order book data directly from the stock exchange. The order book is a snapshot of the market, made up of various “bid” (buy) and “ask” (sell) offers at various prices and sizes. Tick data is fine-grained financial market data, where each datapoint corresponds to a change in the order book. The goal for our project was to find signals to predict short-term changes in price given only this data. Designing an automated procedure to generate these signals is a two-step process. First, we must decide on the search space of possible signals and how we represent them. Second, we need a search algorithm that can effectively explore this space. We chose to represent the signal as a directed acyclic graph, where each node represents a function on its input. We must then decide which functions to allow. We found that fundamental operations like addition and subtraction are not sufficient. Since we are working with time-series data, defining operations that aggregate along the time axis allows us to capture trends in the data. We found that including exponential moving average (EMA) and exponential moving sum (EMS) as node operations substantially improves the quality of the signal. <figure class="wp-block-image aligncenter size-full is-style-default"><img loading="lazy" decoding="async" width="980" height="375" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2.webp" alt="" class="wp-image-273585" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2-480x184.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> We found that <a href="https://en.wikipedia.org/wiki/Genetic_algorithm">genetic algorithms</a> are well-suited for this task given the large amount of parallel computation possible with HRT’s compute resources. Genetic algorithms select the best candidates at each generation and “evolve” them to create a new generation, improving the overall quality of the candidate pool over time. However, a challenge with such an approach is that there can be a tendency to excel at finding a single optimal solution, rather than multiple distinct ones. In this particular problem, we believe that the best possible “signal” would be a combination of multiple signals, so we would prefer that the algorithm doesn’t converge on a best single signal. To address this, we attempted to decorrelate the signals selected at each generation. We expressed this via a “fitness function” that prefers signals that maximize the signal’s correlation with the target returns while minimizing correlation with other signals. <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="800" height="600" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-2.png" alt="" class="wp-image-273586" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-2.png 800w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-2-480x360.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 800px, 100vw" referrerpolicy="no-referrer"></figure> We found that the application of de-correlation pressure does lead to the generation of more diverse signals.  When we combine the top signals in the population to predict the target, we obtain a better model than just using the top signals without the pressure. Overall, we find that this approach can generate predictive and diverse signals. <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <h2 class="wp-block-heading" id="h-speed-up-hyperparameter-search-with-graybox-bayesian-optimization">Speed Up Hyperparameter Search with Graybox Bayesian Optimization</h2> <h3 class="wp-block-heading" id="h-by-ziqian-zhong">By Ziqian Zhong</h3> At HRT, Algo Developers develop new and exciting trading strategies which often have parameters like thresholds that need to be chosen empirically. One way to choose these parameters is by directly optimizing the performance of a strategy on a backtest. As backtests are costly in both time and computation resources, we strive for the best possible results using the fewest evaluations possible. Suppose we have a scoring function that measures the performance of a trading strategy on a backtest. Our goal is then to find the set of parameters that maximize this score. It’s natural to approach this function as an expensive “<a href="https://en.wikipedia.org/wiki/Derivative-free_optimization">blackbox</a>” whose values we can query at any point. <a href="https://en.wikipedia.org/wiki/Bayesian_optimization">Bayesian optimization</a> provides us an efficient way of doing so. The method maintains a belief (“posterior”) on the function: the expected values at each point and corresponding uncertainties, based on which the new points could be picked. For example, we may pick the point that is believed most likely to improve the current maximum (“Probability of Improvement”). After we observe the value at the newly picked point, we then update our belief and continue the process. Blackbox Bayesian optimization is generally more efficient than simpler approaches like grid search and uniform sampling. In some cases, we can go even further if extra “structural” information is available for our scoring function. For example, the final scoring function might be a sum of functions on the same parameters, each of which may be evaluated separately. A relevant example would be if the total profit of a trading strategy could be expressed as the sum of profits over individual days. Using this structure is <a href="https://arxiv.org/abs/2201.00272">playfully called “graybox”</a> optimization, as we’ve opened one layer of the blackbox but do not have full information about the function. Applying the graybox approach to a sum of functions, we model our beliefs about each function that contributes to the sum, not just the value of the sum itself. This more elaborate model can integrate information from partial evaluations. For example, you can reduce uncertainty in the model by exploiting the correlation of one day’s profit with another’s for a similar choice of trading parameters. We ultimately obtain a graybox posterior distribution which we use to decide the next points to sample. <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="980" height="686" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1.webp" alt="" class="wp-image-273587" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1-480x336.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> We evaluated blackbox and graybox Bayesian optimization by comparing their regrets — the difference between the maximum possible score computed with a much larger budget and the discovered maxima (how much value is lost compared to real maxima). Compared to blackbox Bayesian optimization, graybox Bayesian optimization gets us to better solutions earlier and generally produces better results over a range of computation budgets. <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <h2 class="wp-block-heading" id="h-accelerating-quantitative-finance-with-gpu-computation">Accelerating Quantitative Finance with GPU Computation</h2> <h3 class="wp-block-heading" id="h-by-emma-yang">By Emma Yang</h3> HRT has a large amount of compute resources to use for training models, but Algo Developers can always benefit from faster training, whether it’s by training bigger models or trying more model variations in a given amount of time. These models ultimately drive HRT’s trading, so performance engineering is a way to directly affect the profitability of strategies. Some of the models that HRT trains involve “tall & skinny” matrix multiplications. For example, if we have a time series with far more timesteps than features, then performing analyses like linear regression or applying transformations across the time series involves multiplying wide and tall matrices, or tall and small square matrices. If we can optimize this key operation, we should be able to improve overall training speed. We began with investigating whether CUDA’s general matrix multiplication heuristic, provided through <a href="https://developer.nvidia.com/cublas">cuBLAS (CUDA Basic Linear Algebra Subroutine)</a>, picks the optimal algorithm for tall & skinny matrix workloads. cuBLAS uses a heuristic to determine a configuration of algorithm attributes such as the tile size, amount of shared memory, and split-K reduction factor of the matrix multiplication based on the data sizes of the operands. In addition to performing matrix operations, the cuBLAS API allows us to query the heuristic for the top algorithms by runtime for a given workload. My goal was determine whether cuBLAS ever makes sub-optimal choices, given the unique nature of tall & skinny workloads. If algorithm choices were suboptimal, then we would expect anomalies in the linear trend of data size and runtime that could be observed through benchmarking each algorithm over a variety of matrix sizes. In particular, we were interested in matrices where the “tall” dimension was between 100,000 to 10 million and the “skinny” dimension was between 5 and 500. <figure class="wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-5 is-layout-flex wp-block-gallery-is-layout-flex"> <figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="980" height="518" data-id="273588" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-1.png" alt="" class="wp-image-273588" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-1.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-1-480x254.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> <figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="980" height="520" data-id="273589" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05.png" alt="" class="wp-image-273589" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05-480x255.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> </figure> After benchmarking the top 10 algorithms chosen by the cuBLAS heuristic for each data size, we did not find any significant deviations from the expected linear relationship, indicating that cuBLAS is indeed making mostly optimal algorithm choices. Seeing that cuBLAS’ heuristic was performing well in choosing amongst the algorithms in its search space, we looked to literature to find solutions that might be able to beat cuBLAS.  The paper “<a href="https://web.archive.org/web/20220616221224/https://eecs.wsu.edu/~dtao/paper/ICS19-TSM2.pdf">TSM2: optimizing tall-and-skinny matrix-matrix multiplication on GPUs</a>” claims that their custom CUDA kernels for tall & skinny matrix multiplication outperforms cuBLAS in GPU occupancy and runtime on Kepler, Maxwell, Pascal, and Volta GPU architectures. However, when we implemented their proposed algorithm, <a href="https://arxiv.org/abs/2002.03258">TSM2X</a>, and benchmarked TSM2X against cuBLAS for a variety of “tall times small” multiplications, we were not able to replicate the speedups from the paper; we found that TSM2X significantly under-utilizes the GPU and has significantly higher runtimes on Hopper (H100) GPUs. This result shows that performance engineering is an ongoing challenge. As technologically sophisticated firms like HRT invest in this <a href="https://www.bloomberg.com/news/articles/2023-06-21/what-s-the-h100-the-chip-driving-generative-ai-quicktake">latest generation of hardware</a>, we must constantly check that our models and systems are making the most of every bit of computing power they provide. <figure class="wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-6 is-layout-flex wp-block-gallery-is-layout-flex"> <figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="980" height="971" data-id="273591" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-1.webp" alt="" class="wp-image-273591" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-1.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-1-480x476.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> <figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="980" height="943" data-id="273592" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07.webp" alt="" class="wp-image-273592" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07-480x462.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 980px, 100vw" referrerpolicy="no-referrer"></figure> </figure>

2023/9/20

Building a Dependency Graph of Our Python Codebase

<h3 class="wp-block-heading" id="h-introduction">Introduction</h3> In keeping with our roots in high frequency trading, HRT moves fast. As is the case with any metric in engineering, speed has its tradeoffs. Over the last half-decade, HRT saw exponential growth in the size and interconnectedness of its research-oriented Python codebase, owing to a combination of a scrappy engineering culture that generally prizes “good enough” over “perfect,” our collaborative working environment that encourages code sharing between teams, and a period of accelerated growth. As our Python codebase grew to millions of lines, import times increased by an order of magnitude, code changes became more expensive to test, and lint times increased far outside the range of usefulness—we were experiencing the effects of code “tangle.” <h3 class="wp-block-heading" id="h-tangling">Tangling</h3> Code “tangling” is a concept HRTers borrowed from descriptions of the same issue in Dropbox publications1 about their own Python codebase. We call code “tangled” when its dependency graph has many overlapping cycles and unrelated portions of the codebase are coupled through indirect and unintuitive import paths. Tangling can be a problem in any large codebase (including in other languages)! In our experience, tangling affects performance of both runtime imports and static analysis (e.g. mypy), and causes tight coupling which can reduce reliability2. Of these issues, our users find runtime import overhead to be the biggest problem, since it slows down the development iteration loop and wastes CPU time in the datacenter. This is likely more problematic for HRT than most other Python shops, as short-lived Python processes make up a considerable fraction of our computational workload.  The negative effects of tangling can increase quickly—a few misplaced imports and suddenly hundreds of modules are coupled together. The effects of import overhead are magnified by tangling because importing any module in a cycle ends up transitively importing all modules in that cycle (and their dependencies). While some imports are very fast, there are many cases that incur significant overhead. One common way overhead sneaks in is through filesystem access—for example, the now deprecated pkg_resources module crawls the filesystem to locate resources. This process becomes particularly problematic when operating over our Network File System. A further source of computational overhead is the loading of bulky, monolithic C extensions by packages such as pandas and NumPy—and even proprietary extensions. Additionally, some of our pure-Python modules incur a range of costly static initialization steps like detecting environmental features or handling dynamic registration of classes or callbacks. In isolation, each of these introduces manageable import overhead; however, in the most tangled portions of our codebase, the compounding effect can amount to over 30 seconds of import time for most programs. This overhead slows down the development iteration loop, and wastes CPU time in our distributed compute environment. <h3 class="wp-block-heading" id="h-dependency-management">Dependency Management</h3> At a high level, our approach to untangling is to establish and maintain a layered architecture where modules in lower layers don’t import from modules in higher layers. Establishing proper layering helps callers to only import what they need. Ideally, our dependency graph should resemble a directed acyclic graph, where modules are topologically ordered by their assigned layers. However, in practice, some cycles are acceptable so long as they are relatively small and contained to one (sub)package. Transitioning to a better dependency management paradigm requires identifying the current causes of tangling, refactoring the codebase to restructure dependencies, and putting in place dependency validation to avoid future regressions. And all of this work has to be done without pausing development in the codebase! <h3 class="wp-block-heading" id="h-tangle-tools-understanding-the-tangle">Tangle Tools: Understanding the Tangle</h3> Once we understood that tangling underlay many of our developer experience problems, we set out to build a toolkit for analyzing the dependency graph of our codebase. Tangle Tools analyzes Python source code to generate a dependency graph of the entire codebase (nodes correspond to modules and edges to imports). Our users can then utilize the command line and browser interfaces to discover, navigate, and refactor dependencies. A typical untangling workflow involves: <ul class="wp-block-list"> <li>Finding an unwanted transitive dependency</li> <li>Tracing import paths from a source to the unwanted dependency <ul class="wp-block-list"> <li>Computes a flow network between the source and dependency</li> <li>Identifies which edges would reduce flow if removed</li> </ul> </li> <li>Refactoring imports to disconnect the source from the dependency <ul class="wp-block-list"> <li>Utilizes code transformations to automate common refactors (e.g. moving a symbol to a new module and updating existing references)</li> </ul> </li> <li>Putting dependency validation in place to avoid regressions <ul class="wp-block-list"> <li>Users write dependency rules to constrain the dependencies of their modules </li> <li>These rules are checked in our continuous integration pipeline</li> </ul> </li> </ul> <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-3.jpeg" alt="" class="wp-image-273701" style="width:768px" referrerpolicy="no-referrer"></figure> None of this would have been possible without extensive use of open source libraries! We used Python’s built-in ast library to parse our Python source code for imports. This parsing work was parallelized via the stdlib’s powerful concurrent.futures module, allowing us to quickly process thousands of modules. Under the hood we use networkx’s directed graph data structure and extensive library of graph algorithms–we found the flow algorithms to be especially useful. Finally, we use the libcst library to perform automated source code refactors, written as transformations to the concrete syntax tree. <h3 class="wp-block-heading" id="h-conclusion">Conclusion</h3> By developing these dependency management and refactoring workflows we’ve been able to make significant progress on untangling. Previously, finding import dependencies was a slow manual process, and refactoring dependencies was a game of whack-a-mole. Now we can navigate our full dependency graph and find efficient refactors that address the root causes of tangling. <hr class="wp-block-separator alignwide has-text-color has-alpha-channel-opacity has-background is-style-wide" style="background-color:#f2f2f7;color:#f2f2f7"> 1 <a href="https://dropbox.tech/application/our-journey-to-type-checking-4-million-lines-of-python" target="_blank" rel="noreferrer noopener">https://dropbox.tech/application/our-journey-to-type-checking-4-million-lines-of-python</a> 2 <a href="https://en.wikipedia.org/wiki/Coupling_(computer_programming)" target="_blank" rel="noreferrer noopener">https://en.wikipedia.org/wiki/Coupling_(computer_programming)</a>

2023/6/5

Modeling Equities Returns: The Linear Case

<h3 class="wp-block-heading" id="h-introduction">Introduction</h3> Having an accurate high-level overview of market dynamics is crucial in algorithmic trading. At HRT, many trading teams curate their own models to understand risk and performance profiles of their portfolios. Here we’ll dig into a specific class of models called factor models and understand how it encapsulates asset relationships, allows for portfolio optimization, and explains investment performance attribution. <h3 class="wp-block-heading" id="h-modeling">Modeling</h3> The first step to making money and managing risk in the stock market is having an accurate model of where the sources of profit and loss are coming from. We’ll go through some basic modeling assumptions and introduce some ways a portfolio manager might evaluate their positions in the market. For simplicity, say that you observe prices $p_t$ of an asset (say, <code>AAPL</code>) at the end of day $t$. The close-to-close returns are defined as: $$ r_t = \frac{p_t}{p_{t-1}} -1 $$ We have such returns for every asset, every day. Wouldn’t it be nice if these returns could be described by a simple model? We are not aiming for the fanciest model of all (say, the <a href="https://www.youtube.com/watch?v=RXJKdh1KZ0w">turbo</a>–<a href="https://en.wikipedia.org/wiki/Turbo_encabulator">encabulator</a> of returns models), but for something simpler and more reliable, yet not totally uncool and trivial. Here we’ll dive into a type of battle-tested models in this category called factor models. <h3 class="wp-block-heading" id="h-modeling-returns">Modeling Returns</h3> Factor models are linear models that describe the return of a set of assets based on: $$ \mathbf r_t = \mathbf B_t\mathbf f_t + \mathbf \epsilon_t $$ This resembles a good old linear regression, but with different symbols: <ol class="wp-block-list"> <li>$\mathbf r_t \in \mathbb{R}^n$ denotes the assets’ returns;</li> <li>$\mathbf B_t \in \mathbb{R}^{n \times m}$ is the loadings matrix;</li> <li>$\mathbf f_t \in \mathbb{R}^m$ is the vector of factor returns;</li> <li>$\mathbf\epsilon_t \in \mathbb{R}^n$ is the vector of residual (or idiosyncratic) returns.</li> </ol> In practice, for many asset classes the number of assets $n$ is much larger than the number of factors $m$. The loading matrix contains the essential information linking factor returns to asset returns. What is different from a traditional regression is that this is a dynamic relationship. The factor returns $\mathbf f_t$ are random variables, drawn in every period from a multivariate distribution which is assumed to be independent from the previous and next ones. Similarly, the “noise” $\mathbf \epsilon_t$ is random and independent from the factor returns. Here, “idiosyncratic” simply means “asset returns not explained by the loadings and returns.” Each return $\epsilon_{t, i}$ is independent of $\epsilon_{s, j}$ over time (when $s\ne t$) and across assets (when $i\ne j$). We can interpret this as a graphical model from underlying factors linearly impacting each asset’s return with some weight. To simplify notation a bit, let’s assume we’ve fixed $t=T$ and remove the subscript. <figure class="wp-block-image aligncenter size-large is-resized"><img loading="lazy" decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2023/01/gBMuev9-1024x736.jpg" alt="" class="wp-image-18593" width="512" height="368" referrerpolicy="no-referrer"></figure> What is the interpretation of $\mathbf B$? Each column is an asset “characteristic.” For example, $b_{i,1}$ could take the value 1 if the stock $i$ is a tech company, and 0 otherwise. The daily returns of tech companies are positively correlated: <code>GOOG</code>, <code>MSFT</code> and <code>AAPL</code> move together quite a bit. A way to interpret this is to say that the first factor (the “tech sector” factor) is moving up or down, and that tech companies are partially driven by this factor. There are additional factors that can play a role. For example, some companies are more profitable than others, and “profitability” can be another factor. And last, there is the “noise” $\epsilon_i$: something that is very specific to a company, like the returns experienced on the day of an earning announcement. When journalists say “in a crisis all correlations go to 1,” what they really mean is “in a crisis some factors move a lot more than asset-specific returns, and as a result asset returns become more correlated.” This relationship between asset returns and factors may seem intuitive, but it encodes so much subtle information about the market. Let’s see some questions we can answer with this simple but powerful tool! <h3 class="wp-block-heading" id="h-covariance-matrix-estimation">Covariance matrix estimation</h3> Having the covariance matrix of all the assets we can trade is useful. Why? Because it allows us to estimate the volatility of any portfolio. “Volatility” is a finance term to denote “standard deviation of something, at some time horizon.” In our case, imagine that we have invested $w_1=$ $-30M in <code>TSLA</code>, $w_2=$ $10M in <code>AAPL</code> and so on. The vector of our net dollar holdings is $\mathbf w$ and the returns covariance matrix of these assets is $\mathbf\Sigma_r$. The volatility of our portfolio is $\sqrt{\mathbf w’\mathbf\Sigma_r \mathbf w}$. This is the standard deviation of our portfolio’s daily dollar PnL. For many reasons we want to have a first estimate of the uncertainty at short horizon: we need to be adequately funded, and we need to set expectations for our swings. There is a problem: $\mathbf\Sigma_r$ can be really big. Between the Americas, Europe, and Asia, there are easily 10,000 liquid, tradeable assets – that’s 50 million values to estimate. The calculation also requires $n^2$ operations, which is demanding, especially if you have to run this many times per day, or even many times per hour. Factor models to the rescue! If we know the distribution of the factor returns, we could estimate the uncentered empirical covariance matrix of the factors: $\mathbf \Sigma_{\mathbf f}=T^{-1} \sum_{t=1}^T \mathbf f_{t}\mathbf f_t’$. This is a fancy way of estimating variances and covariances of factors, assuming that the factor return means are so small that they do not make a difference in the computation. If we know the idiosyncratic returns, the calculation is even easier. Since these returns are pairwise uncorrelated, the covariance matrix is diagonal with terms $[\mathbf\Sigma_{\epsilon}]_{i,i} = \frac{1}{T} \sum_{t=1}^T\epsilon^2_{t,i}$. Now, we have the covariance matrix of our assets. With a bit of algebra, we get: $$ \mathbf\Sigma_r = \text{Var}(\mathbf r) = \text{Var}(\mathbf B\mathbf f + \mathbf\epsilon) = \text{Var}(\mathbf B\mathbf f) + \text{Var}(\mathbf\epsilon) \mathbf =\mathbf B\mathbf\Sigma_{\mathbf f}\mathbf B’ + \mathbf\Sigma_{\mathbf\epsilon} $$ Storing this matrix takes essentially $mn$ numbers, rather than $n^2$. Since there are typically less than a hundred factors, $m \ll n$, that saves us 2 to 4 orders of magnitude in space! Computing the volatility of our portfolio is also much faster. Since we want to compute $\sqrt{\mathbf w’\mathbf \Sigma_r \mathbf w}$, we could first compute the vector $\mathbf b:=\mathbf B’\mathbf w$, which takes $\sim mn$ operations. The rest is very fast, since computing $\mathbf b’\mathbf\Sigma_{\mathbf f}\mathbf b+ \mathbf w’\mathbf\Sigma_{\mathbf \epsilon}\mathbf w$ takes only $\sim m^2+n$ operations. Most importantly, covariance matrices from factor models often give more accurate volatility estimations than the empirical ones. <a href="https://www.hudsonrivertrading.com/hrtbeat/modeling-equities-returns/#postNotes">[1]</a> <h3 class="wp-block-heading" id="h-optimizing-portfolios-and-performance-ex-ante">Optimizing portfolios and performance ex ante</h3> You have developed a sophisticated prediction system based on deep learning running on a quantum computer located in the brain of the <a href="https://en.wikipedia.org/wiki/World_Turtle">Cosmic Turtle</a>. Every day, it generates for you, and you only, a humble vector $\mathbf \alpha\in\mathbb R^n$. Such vector contains the one-day ahead expected returns of your stock universe. What would you do? One thing you can do is to find a portfolio that maximizes your profitability, measured in units of risk. Otherwise stated: you divide your portfolio’s expected PnL by its volatility, and maximize this ratio. If we ignore a large number of important details like execution costs, financing costs, taxation, multiperiod considerations, regulatory constraints, etc., the problem can be formulated as: $$ \max_{\mathbf w} \frac{\mathbf \alpha’\mathbf w}{\sqrt{\mathbf w’\mathbf\Sigma_r \mathbf w}} $$ The objective function is the Sharpe Ratio of the strategy and is a commonly used objective. It’s not always the most relevant formulation in practice, but it is one of the easiest and most insightful ways to reason about portfolio optimization. How does a factor model help? The objective function $f(\mathbf w)$ does not depend on the unit of currency: if you measure a portfolio in dollars or cents, the Sharpe Ratio stays the same; ditto if you fix your unit of currency, or scale up or down your portfolio holdings by some constant multiplier.<a href="https://www.hudsonrivertrading.com/hrtbeat/modeling-equities-returns/#postNotes">[2]</a> We can fix the denominator to some value–say 1–and maximize the objective, which is now $\mathbf\alpha’\mathbf w$: $$ \begin{align} \max_{\mathbf w}\, & \mathbf \alpha’\mathbf w \\ \text{s.t.} \,& \mathbf w’\mathbf\Sigma_r \mathbf w\le 1 \end{align} $$ The solution is a function of the inverse of the covariance matrix (sometimes called precision matrix): $$ \mathbf w^\star=\frac{\mathbf \Sigma_{\mathbf r}^{-1}\mathbf \alpha}{\sqrt{\mathbf \alpha’\mathbf \Sigma_{\mathbf r}^{-1}\mathbf \alpha}} $$ And the optimal value of the objective function, the Sharpe Ratio, is $\alpha’\mathbf w^\star$ or: $$ \text{Best Sharpe Ratio} ={\sqrt{\mathbf \alpha’\mathbf \Sigma_{\mathbf r}^{-1}\mathbf \alpha}} $$ This highlights the importance of estimating accurately both the covariance matrix for volatility targeting and the precision matrix for Sharpe maximization, which are surprisingly related to each other. But there are more insights to be gained by a factor model. In the formula $\mathbf r_t = \mathbf B_t\mathbf f_t + \mathbf \epsilon_t$ we did not say anything about the expected values of $\mathbf f_t$ and $\mathbf \epsilon_t$. In the remainder, we can express returns as the sum of an expected value term and a zero-mean term: $\mathbf \mu_{\mathbf f} +\tilde{\mathbf f}_t$, $\mathbf \mu_{\mathbf \epsilon} +\tilde{\mathbf \epsilon}_t$. The expected returns of my asset universe are $\mathbf \alpha:=\mathbf B \mathbf \mu_{\mathbf f}+ \mu_{\mathbf \epsilon}$. This suggests that we have some degree of freedom in choosing $\mu_{\mathbf f}$ and $\mu_{\mathbf \epsilon}$. For example, if we add a shift term $\mathbf a$ to $\mu_{\mathbf f}$ and I subtract a term $\mathbf B\mathbf a$ from $\mathbf \mu_{\mathbf \epsilon}$, the expected returns of the assets is unchanged. To remove this indeterminacy, we impose that $\mathbf \mu_{\mathbf \epsilon}$ be orthogonal to the columns of $\mathbf B$; i.e., $\mathbf B’\mathbf \mu_{\mathbf \epsilon}=0$. When we use this orthogonal decomposition, $\mathbf \mu_{\mathbf \epsilon}$ is often called alpha orthogonal and denoted $\mathbf\alpha_{\perp}$, while $\mathbf\alpha_{||}:=\mathbf B\mathbf \mu_{\mathbf f}$ is alpha parallel, because is “parallel” to the factors. Now we replace $\alpha$ with $\mathbf \mu_{\mathbf \epsilon}+\mathbf B \mathbf \mu_{\mathbf f}$ in the Sharpe Ratio formula. To make the formulas a little simpler, we assume that all the idiosyncratic volatilities are identically equal to $\sigma$. After some matrix manipulation<a href="https://www.hudsonrivertrading.com/hrtbeat/modeling-equities-returns/#postNotes">[3]</a> we have a neat formula for the Sharpe Ratio: $$ \begin{align} \mathbf H:= & \mathbf B’\mathbf B\\ \text{Sharpe Ratio} = &\sigma^{-1}\left(|| \mu_{\mathbf \epsilon}||^2 + \mu_{\mathbf f}’ \left[ \mathbf H – \mathbf H(\sigma^2\mathbf\Sigma_{\mathbf f}^{-1} + \mathbf H )^{-1} \mathbf H’ \right] \mu_{\mathbf f} \right)^{1/2} \end{align} $$ Risk-adjusted performance can be decomposed into the contributions of alpha parallel, which is associated to the PnL generated by asset characteristics, and alpha orthogonal: the excess returns of assets that cannot be explained by these same characteristics and is asset-specific. <ul class="wp-block-list"> <li>In the limit of large idiosyncratic risk $\sigma\rightarrow \infty$, factor risk becomes neglible. The model of returns simplifies $\mathbf r_t=\mathbf \alpha+\sigma\tilde{\mathbf\epsilon}_t$. In the formula above, the Sharpe Ratio approaches $\sigma^{-2}||\mathbf\alpha||^2$. It is not surprising that factor terms do not appear in the formula.</li> <li>When idiosyncratic risk is small, SR approaches $\sigma^{-2} ||\mu_{\mathbf \epsilon}||^2$. Notice that $\lim_{\sigma\downarrow 0} SR=\infty$: if there are little excess returns that are not explainable by factors, and if most risk is factor risk, then the Sharpe can be very high indeed. This may explain why a lot of work goes into finding “alpha orthogonal.”</li> </ul> <h3 class="wp-block-heading" id="h-understanding-performance-ex-post">Understanding performance ex post</h3> Controlling risk is nice, but having a profitable strategy is great. A habit of a highly effective trader is to revisit their historic profit and loss (or PnL), and understand its origin. Say that we have a per-period forecast of expected returns $\mathbf \alpha_t = \mathbf B \mu_{\mathbf f, t} + \mu_{\mathbf \epsilon, t}$ and actual returns $\mathbf r_t = \mathbf B (\mu_{\mathbf f, t} + \tilde{\mathbf f}_t) + (\mu_{\mathbf \epsilon, t} + \tilde{\epsilon}_t)$ such that we have expected factor and idio returns $\mu_{\mathbf f,t}$, $\mu_{\mathbf \epsilon,t}$ (note in the previous section we assumed $\alpha$ is stationary).When we look at PnL and attribution to factor returns or idiosyncratic return, we can decompose it as follows: \begin{align} \text{PnL} =& \sum_t\text{PnL}_t \\ = &\sum_t \mathbf w_t’ \mathbf r_t \\ = &\sum_t \mathbf w_t’\mathbf\alpha_t + \sum_t \mathbf w_t'(\mathbf B_t \tilde{\mathbf f}_t) + \sum_t \mathbf w_t’ \tilde{\mathbf \epsilon}_t \\ = & \sum_t \mathbf b_t’\mathbf\mu_{\mathbf f,t} + \sum_t \mathbf w_t’\mathbf \mu_{\mathbf \epsilon,t} +\sum_t \mathbf b_t’\tilde{\mathbf f}_t + \sum_t \mathbf w_t’\tilde{\mathbf \epsilon}_t \\ \approx &\; \text{factor tilt PnL} \\ &+\text{idiosyncratic tilt PnL} \\ &+\text{factor timing PnL}\\ &+\text{idiosyncratic timing PnL}  \end{align} We encountered $\mathbf b_t \in \mathbb{R}^m$ when we estimated portfolio volatility. It is called the factor exposure vector of the portfolio. For each factor, this is the dollar-weighted sum of that factor’s characteristics. A way to see $\mathbf b_t$ is like a low-dimensional sketch of a high-dimensional object – instead of having to list the dollar positions in <code>TSLA</code>, <code>AAPL</code>, and <code>GME</code>, which would be tedious and non-insightful, you can just tell people at a cocktail party that you are long tech (i.e., have more long positions than short positions in tech stocks) and that you are also long meme stocks (or at least, read the room before giving an unpopular answer). The first term is the PnL accrued from leaning(or tilting) into factors that have non-zero expected returns. The second term is similar, but for expected idio returns. The last two terms denote timing skills. Why timing? Because in order to have positive timing PnL we need to guess the signs right – we should have positive exposure to a factor with positive returns and negative exposures otherwise. In other words, the covariance covariance between factor exposures and factor returns is positive. Here is a precise statement for factor timing (idio timing can also be stated similarly): \begin{align} \sum_t \mathbf b_t’\tilde{\mathbf f}_t =& \sum_t \sum_i b_{t,i}  \tilde{f}_{t,i}\\  =& T\sum_i\frac{1}{T} ( \sum_t (b_{t,i} \tilde{ f}_{t,i} -E[b_{t,i}] E[\tilde f_{t,i}]))    \\ =&  T\sum_i \text{cov}  (\mathbf b_{\cdot,i},  \tilde{\mathbf f}_{\cdot,i}) \end{align} The factor timing component is the sum of the time-series covariances of exposures and centered factor returns. It should not come as a surprise that the timing PnL is almost always negligible in the long run unless you have the cosmic turtle on your side. <h3 class="wp-block-heading" id="further-reading"> Further Reading</h3> We have scratched only the surface of this topic. For example, we have not at all covered the estimation problem, and the connection between generalized linear models and alpha research (aka The Brain of The Cosmic Turtle). For the interested reader, here are a few useful references, in increasing order of complexity: <a href="https://www.amazon.com/Advanced-Portfolio-Management-Fundamental-Investors/dp/1119789796/ref=sr_1_1?crid=QNYERCFXY0MX&keywords=advanced+portfolio+management+by+paleologo&qid=1667536648&qu=eyJxc2MiOiIwLjI2IiwicXNhIjoiMC4xNyIsInFzcCI6IjAuMDAifQ%3D%3D&s=books&sprefix=advanced+por%2Cstripbooks%2C85&sr=1-1">Advanced Portfolio Management</a> by Paleologo. Wiley (2021) <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1024709">Factor Models of Asset Returns</a> by Connor and Korajczyk. Encyclopedia of Quantitative Finance(2010) <a href="https://www.amazon.com/Portfolio-Risk-Analysis-Gregory-Connor/dp/0691128286/ref=sr_1_2?crid=235HM9CGARY2Z&keywords=portfolio+risk+analysis&qid=1667536488&qu=eyJxc2MiOiIwLjgzIiwicXNhIjoiMC4wMCIsInFzcCI6IjAuMDAifQ%3D%3D&s=books&sprefix=portfolio+risk+analysis%2Cstripbooks%2C85&sr=1-2&ufe=app_do%3Aamzn1.fos.f5122f16-c3e8-4386-bf32-63e904010ad0">Portfolio Risk Analysis</a> by Connor, Goldberg and Korajczyk. Princeton University Press (2010) <a href="https://www.amazon.com/Statistical-Foundation-Monographs-Statistics-Probability/dp/1466510846">Statistical Foundations of Data Science</a> by Fan, Li, Zhang, Zou. CRC Press (2020) <a href="https://www.amazon.com/Spectral-Analysis-Dimensional-Matrices-Statistics/dp/1441906606">Spectral Analysis of Large Dimensional Random Matrices</a> by Bai and Silverstein. Springer (2017) <h3 class="wp-block-heading" id="postNotes">Notes</h3> <ol class="wp-block-list"> <li>Here is a pathological example: Choose a test portfolio $\textbf w$ that is orthogonal to each vector of historical returns $\textbf w’\textbf r_t=0$ for all $t$. Such portfolio is guaranteed to exist because the number of periods used from estimation is 250-500), much smaller than the number of assets (>1000). It follows that the predicted vol using the empirical covariance matrix is 0, because $\textbf w (\sum_t T^{-1}\textbf r_t\textbf r_t’) \textbf w=0$. This a serious underestimation of risk. <img loading="lazy" decoding="async" width="15" height="15" class="wp-image-19275" style="width: 15px;" src="https://www.hudsonrivertrading.com/wp-content/uploads/2023/02/21a9-fe0f@2x.png" alt="" referrerpolicy="no-referrer"> </li> <li>The Sharpe Ratio is homogeneous of degree 0: for any positive number $a$, it is $f(a\mathbf w)=f(\mathbf w)$. <img loading="lazy" decoding="async" width="15" height="15" class="wp-image-19275" style="width: 15px;" src="https://www.hudsonrivertrading.com/wp-content/uploads/2023/02/21a9-fe0f@2x.png" alt="" referrerpolicy="no-referrer"> </li> <li>To derive the result, use <a href="https://en.wikipedia.org/wiki/Woodbury_matrix_identity">Woodbury’s identity</a> for matrix inversion. Note that even if individual idiosyncratic volatilities differ so that $\mathbf \Sigma_\epsilon \neq \sigma^2I$, this still provides a computationally efficient way of inverting the estimated covariance matrix as long as $\mathbf \Sigma_\epsilon$ is diagonal. <img loading="lazy" decoding="async" width="15" height="15" class="wp-image-19275" style="width: 15px;" src="https://www.hudsonrivertrading.com/wp-content/uploads/2023/02/21a9-fe0f@2x.png" alt="" referrerpolicy="no-referrer"></li> </ol>

2023/3/9

Optimizing UX/UI Design for Trading at HRT

It might seem counterintuitive for a trading firm with a heavy emphasis on automation to spend resources on user interfaces; however, UX/UI can have a large impact on even fully automated trading. While we try to automate as much as we can, UX/UI is critical in the cases where we need a human in the loop. At HRT, engineers and researchers rely on UIs to monitor live trading performance, interact with various automated processes/systems, and participate in markets that are not fully electronic. As the firm expands to new types of trading, UX/UI design has become increasingly important. Designing at HRT has some unique advantages and some unique challenges. HRT’s culture is unusually open for a trading firm — almost all employees have access to all of HRT’s code and data. This makes it easy to run design concepts across my user base, since they are all generally aware of the same systems. People at HRT are typically very interested in explaining how complex systems work, which is important because some of the systems we are attempting to visualize can be very intricate, especially in comparison to a consumer UX. So while the job can be challenging, the culture goes a long way in allowing me to get some impactful results – let’s examine one project. <h3 class="wp-block-heading" id="h-task-watcher">Task Watcher </h3> Task Watcher is a tool for monitoring the status of distributed computations. HRT runs a lot of parallelized calculations in our research and production trading, so it’s important for researchers to have visibility into the progress of their calculations, any failures that emerge, and efficiency metrics such as runtime and memory usage. Our Task Watcher UI provides users a point-and-click interface for quickly accessing this information. Prior to my work, the original UI looked like this: <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-2.jpeg" alt="" class="wp-image-273574" style="width:768px;height:424px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">The original tool showed lots of information on one screen, making it difficult to find errors.</figcaption></figure> To achieve user-centered layouts, I adopt the following principles throughout my design process: <h3 class="wp-block-heading" id="h-principle-1-design-with-a-deep-understanding-of-user-goals">Principle #1. Design with a deep understanding of user goals</h3> Before a single line of code is written, fully understand what the users are trying to achieve, rather than simply implementing feature requests. One of the benefits of UI development at HRT is sitting right next to my customers — the developers, programmers, and researchers using my tools. Unlike B2C companies, who rely on competitive analysis, market studies, and user personas to gather customer insight, my understanding of my user base comes from direct and ongoing conversations with peers. After conducting interviews with a cross-section of users, I came to two conclusions: <ol class="wp-block-list"> <li>Too many tabs made it difficult to find features. </li> <li>Screens were built as a stack of features based on a list of user requests.</li> </ol> As a result, workflows didn’t follow the user’s mental model, which meant that only 20% of features were actually being used. For example, even though the same information was available in the Task Watcher UI, users would use the old terminal interface to view the error summary because they weren’t aware it was hidden away in a tab at the bottom. One technique I used to figure this out was <a href="https://www.nngroup.com/articles/card-sorting-definition/">card sorting</a>, which lets me visualize how all of the features fall into different categories in the user’s mental model. In the following example, there are 5 sub-tabs (Summary, Error, Report, Log, and Plexus) and over 80 fields. During my interviews with users, very few actually knew what these fields were for. After running some <a href="https://www.nngroup.com/articles/task-analysis/">task analysis</a>, we discovered users had three goals: (1) Monitor activities, (2) Find errors, (3) Debug.  <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1.jpeg" alt="" class="wp-image-273575" style="width:641px;height:768px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">The previous layout included over 80 fields in the bottom tabs.</figcaption></figure> <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03-1.jpeg" alt="" class="wp-image-273576" style="width:768px;height:321px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">After card sorting, I found three goals and redesigned the structures.</figcaption></figure> During my interviews with users, this technique guided my exploration of which existing features users were familiar with and helped me identify why they used (or did not use) any of them. <h3 class="wp-block-heading" id="h-principle-2-tailored-controls-are-key-to-making-actions-faster">Principle #2. Tailored controls are key to making actions faster</h3> When I design, I think through scenarios for the most heavily used controls to determine best practices. In Task Watcher, it was common for users to manually monitor runtimes to find problematic or slow tasks. This was laborious and error-prone. The new UI includes a screen that has a visual display of task runtimes, highlighting outliers so problematic tasks can be easily found. <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-1.jpeg" alt="" class="wp-image-273577" style="width:768px;height:167px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">In the old Task Watcher, users read run time and memory from a table.</figcaption></figure> <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05.jpeg" alt="" class="wp-image-273578" style="width:768px;height:254px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">The redesigned chart provides an intuitive way for users to analyze job performance across multiple runs. In this example, bar length indicates runtime and bar color highlights pending tasks (yellow) and errors (red).</figcaption></figure> <h3 class="wp-block-heading" id="h-principle-3-establish-a-design-system-to-ensure-visual-consistency">Principle #3. Establish a design system to ensure visual consistency</h3> Standardizing design language also helps me build a seamless trading experience across multiple UIs and multiple parts of a single UI. In addition, consistency makes it easier for users to understand and navigate the interface. When features are added without regard for an existing design system, developers have to make choices that don’t necessarily improve the user experience. For example, the previous design used green text to indicate success, which made sense in isolation, but it was hard to read against some of the existing background colors. There are many tools online that can help you decide if certain color combinations result in readable text. We use the <a href="https://coolors.co/contrast-checker/112a46-acc8e5">Color Contrast Checker from Coolors</a>. Creating a complete design system allows us to think through features in aggregate, ensuring that everything remains readable. For Task Watcher, I designed a dark theme template with specifically chosen colors for each type of task progress or category of information. Redesigned monitoring screen:  <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06.jpeg" alt="" class="wp-image-273579" style="width:768px;height:487px" referrerpolicy="no-referrer"></figure> <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07.jpeg" alt="" class="wp-image-273580" style="width:768px;height:277px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">By using the full real estate of the screen, we can display important information in a high-level summary; less relevant information is pushed off the monitoring view.</figcaption></figure> Redesigned detail screen: <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/08.jpeg" alt="" class="wp-image-273581" style="width:768px;height:487px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">In addition to simplifying the initial view, I re-designed a screen to highlight further details that users might need.</figcaption></figure> <h3 class="wp-block-heading" id="h-principle-4-continuously-gather-and-incorporate-user-feedback">Principle #4. Continuously gather and incorporate user feedback.</h3> <figure class="wp-block-image alignright size-full is-style-default"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/09-1.jpeg" alt="" class="wp-image-273582" referrerpolicy="no-referrer"></figure> As I roll out new designs, I continue to collect real-time feedback from users to address early roadblocks. Working in close proximity to my user base enables me to troubleshoot issues as soon as they are identified and implement solutions under quick turnaround times. The design process for Task Watcherwas continually <a href="https://xd.adobe.com/ideas/process/user-testing/process-user-testing-iterative-usability-testing-best-practices/">iterated on</a> and modified according to user feedback or technical limitations. For example, the display for currently running tasks initially included a progress bar. However, we realized that it wasn’t possible for every type of task to report its progress, so we eliminated this feature from the display.  <h3 class="wp-block-heading" id="h-conclusion">Conclusion</h3> As HRT embraces increasingly complex user interactions, we must prioritize an efficient design process that begins with a dialogue with our users. This starts with my comprehensive understanding of their thought process in order to distill lists of requirements into an essential UX that reflects the user’s real intent, whether that’s monitoring a complex distributed task tree, monitoring complex risk metrics, or executing critical trades. When I first face a new UX, the complexity can be intimidating, but by applying UX fundamentals and spending a lot of time with my users, I’m able to achieve results I’m proud of that improve our work as a firm.

2023/1/27

Intern Spotlight: 2022 HRT AI Labs Summer Projects

<h3 class="wp-block-heading" id="h-introduction">Introduction</h3> HAIL (HRT AI Labs) was lucky to host several talented Algo Developer interns during the HRT 2022 Algo Summer Internship program. These interns tackled the topic of “artificial intelligence in trading” from a broad variety of perspectives, with some fascinating results obtained in the short five weeks they spent on a rotation with our team. In this blog post, we’re highlighting the work of four interns across three topics: <ul class="wp-block-list"> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-hrt-ai-labs-summer-projects/#applylargelanguage">Applying Large Language Models to Financial News, by Max Emerling</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-hrt-ai-labs-summer-projects/#h-signal-architecture-search">Signal Architecture Search, by Louie Putterman</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-hrt-ai-labs-summer-projects/#h-transformers-for-market-data">Transformers for Market Data, by Sean Mann and Jerry Mao</a></li> </ul> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="applylargelanguage">Applying Large Language Models to Financial News</h2> <h3 class="wp-block-heading" id="h-by-max-emerling">By Max Emerling</h3> Large language models (LLMs) like <a href="https://openai.com/api/">OpenAI’s GPT-3</a> have demonstrated impressive results in Natural Language Processing over the past couple of years. It is debatable if these models are truly “smart” or getting us closer to true “Artificial General Intelligence,” but one thing is for sure: they know a lot. Baked within the hundreds-of-billion parameters in these models is an impressive ability to perform tasks that range from financial analysis to writing romance stories. The former, as you might expect, is important to us here at HRT, so it would be nice to find a way to extract structured information from these models and the unstructured data they are trained on. It may help to think of an LLM as a big, messy database. In a way, prompting an LLM is similar to querying for a certain subset of its training data. However, a very simple change in the style or wording of your prompt can drastically affect the quality of your results. For example, if we prompt an LLM with “Database company name: MARATHON DIGITAL HOLDINGS INC. Competitors: “then it will complete the “sentence” incorrectly with “Bitcoin, Ethereum, Litecoin.” Perhaps the name being in all-caps is the issue? Changing the prompt to include “Simple company name: Marathon Digital Holdings” gives us a better completion: “Bitmain, Canaan, Halong Mining” (Marathon Digital Holdings owns and operates cryptocurrency mining hardware). While impressive, using LLMs to trade is more complex. To start, we need a market moving source of text for LLMs to analyze. One natural fit here is news headlines – short strings of text that often lead huge market moves. Still, there are many ways we could make a trade. Let’s start simple – given a set of known market-moving headlines, can an LLM predict the direction the market moved? We found that if we just give the headline and ask if it is positive or negative, the results are not much better than tossing a coin. However, if we do a little “prompt engineering” and ask the model to reason about the headline, we get better results: <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="640" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-1.png" alt="" class="wp-image-273569" style="width:768px;height:480px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-1.png 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-1-980x613.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-1-480x300.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> We also find results consistent with other domains, namely that the larger the LLM, the better the results. Additionally, the more examples we provide the model of positive and negative headlines before prompting it on a new headline, the better it does, and the effect compounds with model size: <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="640" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1.webp" alt="" class="wp-image-273570" style="width:768px;height:480px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1.webp 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1-980x613.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1-480x300.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> Clearly this is just the start, and there are many similar (and very different) experiments we can try to keep boosting accuracy. Even these simple examples show the exciting capabilities of LLMs. While you can’t necessarily ask them anything and expect a useful out-of-the-box answer, a bit of creativity and formulaic tinkering can lead to some very promising results. <h2 class="wp-block-heading" id="h-signal-architecture-search">Signal Architecture Search</h2> <h3 class="wp-block-heading" id="h-by-louie-putterman">By Louie Putterman</h3> At the core of many trading strategies is the notion of a “signal.” Signals are a function which take in some features of the world and output a value which is correlated with the future movements of the market. Basic signals can be something like the sentiment of words in news headlines or the relative strength of an earnings report to the predicted values. Often, signals are guided by human intuition and are thus necessarily limited in their scope and complexity. In this project we explored what happens when we equip evolutionary algorithms with rich data, and train them to discover signals. In financial markets, data is everywhere. However, if we are going to learn and build functions on this data, we must require that it has a constrained shape. Fortunately, the market itself is already organized into something called an order book. The order book contains two sides, the bid (buying) and the ask (selling), where each side is then sorted into levels. Each level contains a price (of the corresponding ask or buy) and a size. This data is sorted by price, and by limiting ourselves to only the top few levels, we can be guaranteed to have relevant, informative, and well-shaped data. Next, we have to decide what kinds of functions we want to learn. We want them to be expressive enough to capture complex trends, but not so complex that their latency becomes an issue. A simple choice for expressing this kind of computation is with the four common operations (+-*/) and an identity operation. With this function vocabulary, we can quite easily generate a random computation graph where each node is an operation. Each node has two input edges corresponding to the features that are being fed in. The output from any node is the result of applying the nodes’ operation to its input features. Finally, the question becomes how we can evolve these functions to learn signals. As in any evolutionary algorithm, we need to decide how to initialize our population, decide who survives each round, and create the next round of offspring. Fortunately, our fitness function is quite straightforward – just the correlation between the function’s output and the future behavior of the asset. Initializing the population is similarly straightforward, as we can simply randomly generate input edges and operations in a computation graph. Finally, to generate the next round of offspring, we introduce the notion of a mutation – simply put, an alteration of the operation or input edges to any specific node. With all of this infrastructure, we are then able to evolve computation graphs to find signals in data. Experimentally, these signals generalized to our testing data and outperformed our baseline signals which accounted for market imbalance. <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="512" height="512" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03.gif" alt="" class="wp-image-273571" referrerpolicy="no-referrer"></figure> Above, we can see the evolutionary process of one of these computation graphs. The colors of nodes correspond to the strength of the correlation of the output, and the red path shows which operations are used on the way to the strongest output. <h2 class="wp-block-heading" id="h-transformers-for-market-data">Transformers for Market Data</h2> <h3 class="wp-block-heading" id="h-by-sean-mann-and-jerry-mao">By Sean Mann and Jerry Mao</h3> Transformers, a type of neural network, have been all the craze in natural language processing recently – as discussed above, models like GPT-3 have been producing astonishingly good results across a wide range of language-related tasks, such as text generation and question answering. These model architectures have impressive abilities to learn complex relationships in sequential, possibly non-numeric, data. It’s natural to wonder whether transformers are also well-suited to learning structure in financial data. We attempted to understand this at two different temporal scales: tick-by-tick order book events for a single instrument, and minute-to-minute returns across a broad cross-section of instruments. <h3 class="wp-block-heading" id="h-tick-by-tick-data">Tick-by-tick Data</h3> Tick data is the most fine-grained data available from financial markets. Various market participants in the market place and cancel orders to reflect their updated beliefs and desires, and sometimes finding a counterparty with a mutually agreed upon price to trade with. The order book represents the state of all active orders, so being able to predict future order books, or somewhat equivalently future orders, would be immensely useful for trading. We first need to represent the order book in a way that a neural network can understand. A big challenge for this project was representing an order book accurately and succinctly as a vector of numbers, while capturing its discrete aspects such as discrete prices and order sizes. We found that there are intricate relationships between representation and the model’s ability to learn and extrapolate. Once a good representation scheme is identified, the issue of model architecture and formulation is also an open-ended question: if we represent the order book as a vector of numbers, then predicting a future order book is a simple regression problem, but this framing can make it difficult for the model to represent its own uncertainty about the future, versus probabilities over categories in a classification problem. We investigated integrating transformer architectures with variational autoencoders (VAE) and found we were able to sample diverse sets of future trajectories conditioned on real market data. <h3 class="wp-block-heading" id="h-cross-sectional-return-data">Cross-sectional Return Data</h3> A more broadly familiar form of market data is the time series of the price of each asset, which we can equivalently view as a sequence of returns over time. We put aside the classic quantitative finance task of forecasting future stock prices in favor of understanding what types of behaviors are exhibited by price/return trajectories generated from transformer models, especially when generating for a broad cross-section of instruments. We considered models with GPT-style neural networks as their core module, with varying embedding algorithms and decoder modules that generate prices. In particular, we feed in historical return data for many instruments and expect to receive future return data. We experiment with a variety of regression objectives to train our generative models, in addition to quantizing stock returns and using classification objectives. We compared these with simpler baseline models like random walks and attention-free linear autoregressors. We found that our regression-based models generally performed better at capturing symbol-to-symbol correlations, with correlation matrices on future returns more closely approximate the ground truth matrix. However, the trajectories produced by these models tended to become “degenerate” quickly, as the regression objectives encouraged predictions near the mean; that is, almost-zero returns. In this sense, any variance in the generated price trajectories would be predominantly noise. On the other hand, our model trained on the quantized dataset did not suffer from such an issue. One explanation may be that the classification objective removes any strong prior on the shape of the return distribution, unlike regression. This enabled more diverse distributions from which generative price paths could be sampled.  <figure class="wp-block-image aligncenter size-full"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04.jpeg" alt="" class="wp-image-273572" referrerpolicy="no-referrer"></figure> There is yet so much more to be explored, such as understanding what the attention modules are learning, and understanding what types of market dynamics the model finds more difficult to learn.

2022/12/7

Low Latency Optimization: Using Huge Pages on Linux (Part 2)

Welcome to the second part of our series on improving latency through the use of huge pages! While the <a href="https://www.hudsonrivertrading.com/hrtbeat/low-latency-optimization-part-1/" target="_blank" rel="noreferrer noopener">first installment</a> was focused on potential hardware benefits, this post will hone in on the practicality of using huge pages to run software – and the most important caveats to be aware of. <h3 class="wp-block-heading" id="h-why-is-huge-page-allocation-tricky">Why is huge page allocation tricky?</h3> One might wonder why this post is even necessary — why can’t one just edit some config file and have the operating system allocate huge pages? It is important to understand that even after configuring Linux to support allocating some 2MiB pages, the OS still uses 4KiB pages for most of its internal workings. Therefore, allocating a huge page is not just about finding one free page; the OS must identify 512 contiguous 4KiB pages, starting at an address that’s a multiple of 2MiB, to create one huge page. This can be quite involved. <h3 class="wp-block-heading" id="h-memory-fragmentation">Memory fragmentation</h3> If there is a lot of free memory on the machine, creating a new huge page is usually not a problem. But when free memory becomes constrained, memory often gets fragmented and there might not be any suitable set of 512 contiguous physical free 4KiB pages. The OS must then defragment some of the memory. Typically, the memory used by programs will be moved from one physical page to another. The programs’ data will, of course, not change — it will simply be stored in a different part of the RAM. However, moving a program’s memory involves suspending it for some amount of time if it’s currently running. Therefore, defragmenting can be quite disruptive1 and make some huge page allocations expensive.  1 Recent Linux versions <a href="https://lwn.net/Articles/817905/" target="_blank" rel="noreferrer noopener">proactively</a> defragment to “smooth out” the latency spikes. <h3 class="wp-block-heading" id="h-transparent-huge-pages">Transparent huge pages</h3> One very easy way to get the memory of long running processes to be backed by huge pages is to use a facility called <a href="https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html" target="_blank" rel="noreferrer noopener">Transparent Huge Pages</a> (THP). It was introduced in Linux 2.6.38 over 10 years ago, so it is commonly available. When THP is in-use, the operating system will replace the physical backing of processes with huge pages on its own when it deems it possible/necessary2. Linux supports two THP modes: the always mode, and the madvise mode. In the always mode, most memory allocated by programs is eligible to be backed by huge pages. In the madvise mode, programs must explicitly tell the kernel to use huge pages for parts of their memory using the <a href="https://man7.org/linux/man-pages/man2/madvise.2.html" target="_blank" rel="noreferrer noopener">madvise</a> system call. The always mode is by far the simplest way to access the benefits of huge pages. However, programs have no control over which memory might end up being backed by huge pages and when. In the madvise mode, they can decide which memory could be promoted, but this requires modifying the source code or inserting madvise calls at runtime (using a <a href="https://man7.org/linux/man-pages/man8/ld.so.8.html" target="_blank" rel="noreferrer noopener">LD_PRELOAD</a> library for example). THP is a good choice for a variety of workloads, especially those with long-running memory-hungry programs that can tolerate some occasional latency spikes in exchange for lower latency during most memory accesses. One could also imagine turning on THP during a program’s initialization hoping to maximize the huge page usage, then turning it off to avoid latency spikes. Note that Linux will break down THP transparently if necessary (under memory pressure for example), and that can also be a source of additional latency. The “magic” of THP comes at the expense of complexity in the operating system; while this feature has existed for a long time, it took a few years to become stable enough for production use. Therefore, a fairly recent kernel is recommended when trying it out. It should also be noted that ‘always mode’ of THP is not suitable for all programs and <a href="https://www.couchbase.com/blog/often-overlooked-linux-os-tweaks/" target="_blank" rel="noreferrer noopener">some</a> <a href="https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/" target="_blank" rel="noreferrer noopener">prominent</a> <a href="https://www.percona.com/blog/2019/03/06/settling-the-myth-of-transparent-hugepages-for-databases/" target="_blank" rel="noreferrer noopener">open</a> <a href="https://github.com/redis/redis/blob/unstable/src/latency.c#L368" target="_blank" rel="noreferrer noopener">source</a> software recommends disabling THP altogether. 2 Some <a href="https://mazzo.li/posts/check-huge-page.html" target="_blank" rel="noreferrer noopener">tricks</a> can be used to encourage the kernel to allocate huge pages <h3 class="wp-block-heading" id="h-hugetlbfs">Hugetlbfs</h3> The other way to allocate huge pages is through a pseudo filesystem called <a href="https://www.kernel.org/doc/html/v4.18/admin-guide/mm/hugetlbpage.html" target="_blank" rel="noreferrer noopener">hugetlbfs</a>.  Some documents also refer to this facility as explicit huge pages. Unlike THP, hugetlbfs uses a specific pool of huge pages. The minimum size of the pool is pre-allocated, while the maximum is a hard upper limit. If pages are available in the pool, allocating a huge page is as fast as allocating a 4KiB page. When no free page is available in the pool but the maximum has not been reached, Linux will attempt to create a fresh new huge page. Memory in the hugetlbfs pool must be explicitly requested by programs and is unavailable to the operating system for its own use. Therefore, sizing the pool appropriately is of the utmost importance in order not to starve the operating system and programs that will not be using huge pages. On the flip side, hugetlbfs pages cannot be swapped out to the disk (but can be <a href="https://lwn.net/Articles/849860/" target="_blank" rel="noreferrer noopener">moved</a>). Be aware that hugetlbfs has an initially surprising and important caveat: when a program requests a new huge page but the pool is already maxed out, the OS will send <a href="https://man7.org/linux/man-pages/man7/signal.7.html" target="_blank" rel="noreferrer noopener">SIGBUS</a> to the process which, if unhandled, will kill it. To avoid this problem, hugetlbfs will by default reserve pages in its pool during allocation (i.e during <a href="https://man7.org/linux/man-pages/man2/mmap.2.html" target="_blank" rel="noreferrer noopener">mmap()</a>). But it’s still possible to unexpectedly run out of pages during <a href="https://en.wikipedia.org/wiki/Copy-on-write" target="_blank" rel="noreferrer noopener">copy-on-write</a> (after a <a href="https://man7.org/linux/man-pages/man2/fork.2.html" target="_blank" rel="noreferrer noopener">fork()</a>, for example3). Therefore, as a rule of a thumb, it’s safer not to use hugetlbfs in a process that needs to spawn child processes. 3Only the child process can receive SIGBUS after a copy-on-write fault. There are other situations where huge pages allocation can fail and trigger a SIGBUS but they are more rare such as specific <a href="https://man7.org/linux/man-pages/man2/set_mempolicy.2.html" target="_blank" rel="noreferrer noopener">NUMA policy</a> or <a href="https://man7.org/linux/man-pages/man2/mmap.2.html" target="_blank" rel="noreferrer noopener">MAP_NORESERVE</a> mappings. <h3 class="wp-block-heading" id="h-support-for-huge-page-allocation-in-software">Support for huge page allocation in software</h3> Generally, most of the memory usage of programs come from what’s called anonymous memory. It’s obtained by calling a memory allocator. Most of the high performance memory allocators (<a href="http://jemalloc.net/" target="_blank" rel="noreferrer noopener">jemalloc</a>, <a href="https://google.github.io/tcmalloc/temeraire.html" target="_blank" rel="noreferrer noopener">tcmalloc</a>, <a href="https://microsoft.github.io/mimalloc/environment.html" target="_blank" rel="noreferrer noopener">mimalloc</a>, etc.) support allocating 2MiB pages backed by either THP, hugetlbfs, or both. Some programs also natively support allocating huge pages using hugetlbfs for their memory, examples include the <a href="https://www.oracle.com/java/technologies/javase/largememory-pages.html" target="_blank" rel="noreferrer noopener">Java VM</a> and <a href="https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-HUGE-PAGES" target="_blank" rel="noreferrer noopener">postgresql</a>. One glaring exception used to be the <a href="https://www.gnu.org/software/libc/" target="_blank" rel="noreferrer noopener">glibc</a> allocator, which is, for most, the default allocator on Linux. Fortunately, glibc 2.35 introduced native support for allocating hugetlbfs pages controlled by the <a href="https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html" target="_blank" rel="noreferrer noopener">glibc.malloc.hugetlb</a> tunable. When using earlier versions of glibc (<2.344), a <a href="https://github.com/libhugetlbfs/libhugetlbfs" target="_blank" rel="noreferrer noopener">library</a> can be plugged into the main heap segment allocator5 to use hugetlbfs pages or THP in madvise mode.  4Glibc 2.34 removed support for external libraries to provide huge pages but does not have native support for huge pages. 5Additional segments allocated by the glibc allocator do not use morecore() and therefore will use 4K mappings. <h3 class="wp-block-heading" id="h-running-programs-from-huge-pages">Running programs from huge pages</h3> A <a href="https://github.com/libhugetlbfs/libhugetlbfs" target="_blank" rel="noreferrer noopener">few</a> <a href="https://github.com/intel/iodlr" target="_blank" rel="noreferrer noopener">projects</a> even support running programs out of hugetlbfs pages. These work by copying the program code onto huge pages during initialization. Note that some tools might not understand what happened and struggle to realize that this memory is coming from a program. Therefore, it is recommended to carefully measure your workload (<a href="https://web.archive.org/web/20230921193747/https://www.intel.com/content/dam/develop/external/us/en/documents/run-perf-opt-bp-large-code-pages-q1update.pdf" target="_blank" rel="noreferrer noopener">this paper</a> has good examples) before looking into running software out of huge pages in order to verify that it does not break your debugging pipeline. Another drawback to consider is that running programs out of anonymous pages prevents the OS from sharing the pages containing the code between multiple instances. This can significantly increase the memory footprint of running programs and induce L3 thrashing. <a href="https://github.com/libhugetlbfs/libhugetlbfs" target="_blank" rel="noreferrer noopener">Libhugetlbfs</a> provides a way to share the remapped segments but it comes with its own set of <a href="https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO" target="_blank" rel="noreferrer noopener">caveats</a>. Another option using THP is the <a href="https://man7.org/linux/man-pages/man5/tmpfs.5.html" target="_blank" rel="noreferrer noopener">tmpfs</a> filesystem mounted with the ‘huge’ option. Files can be copied to this filesystem before running and the program’s instructions will reside on a  THP-enabled region while running6.  Shared libraries can also be copied and loaded from this filesystem. 6Note that before Linux 5.7 a specific configuration option needs to be enabled for this option to be available (CONFIG_TRANSPARENT_HUGE_PAGECACHE) <h3 class="wp-block-heading" id="h-fine-tuning-the-tlb-usage">Fine tuning the TLB usage</h3> As <a href="https://www.hudsonrivertrading.com/hrtbeat/low-latency-optimization-part-1/" target="_blank" rel="noreferrer noopener">part 1</a> explained, making better use of the TLB is the main benefit of huge pages. Until now, we have described the TLB as an indiscriminate array of entries that store memory translations. However, like the <a href="https://en.wikipedia.org/wiki/CPU_cache" target="_blank" rel="noreferrer noopener">CPU caches</a>, the TLB is split into instructions and code entries, and uses multiple levels.  This could affect software performance and should be taken into account. On x86 CPUs, the TLB has 2 levels. The first level — which has the lowest latency — is split into an instruction specific TLB (iTLB) and a data TLB (dTLB). Depending on the specific CPU, each level will have specific 4KiB, 2MiB or 1GiB slots in the TLB (<a href="http://www.etallen.com/cpuid.html" target="_blank" rel="noreferrer noopener">cpuid</a> can be used to retrieve this information). Here is the output for a <a href="https://en.wikichip.org/wiki/intel/microarchitectures/cascade_lake" target="_blank" rel="noreferrer noopener">Cascade Lake</a> CPU: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)"><span role="button" tabindex="0" data-code="cache and TLB information (2): 0x63: data TLB: 2M/4M pages, 4-way, 32 entries data TLB: 1G pages, 4-way, 4 entries 0x03: data TLB: 4K pages, 4-way, 64 entries 0x76: instruction TLB: 2M/4M pages, fully, 8 entries 0xb6: instruction TLB: 4K, 8-way, 128 entries 0xc3: L2 TLB: 4K/2M pages, 6-way, 1536 entries" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>cache and TLB information (2): 0x63: data TLB: 2M/4M pages, 4-way, 32 entries data TLB: 1G pages, 4-way, 4 entries 0x03: data TLB: 4K pages, 4-way, 64 entries 0x76: instruction TLB: 2M/4M pages, fully, 8 entries 0xb6: instruction TLB: 4K, 8-way, 128 entries 0xc3: L2 TLB: 4K/2M pages, 6-way, 1536 entries</code></pre></div> On this CPU, each core has a L1 dTLB with 32 entries for 2MiB pages, 64 entries for 4KiB pages and 4 entries for 1GiB pages. The L2 TLB has 1536 entries shared by instruction and data 4KiB and 2MiB pages. Therefore to get the lowest latency, keeping some hot pages as 4KiB might be required. <h3 class="wp-block-heading" id="h-conclusion">Conclusion</h3> In this second post concerning huge pages, we covered the practical use of huge pages and some of the caveats to keep in mind. Various kernel facilities and libraries allow backing both anonymous and file-backed code and data mappings with huge pages, but each have their idiosyncrasies. We recommend carefully reviewing the benefits and drawbacks of both hugetlbfs and THP. As both come with their own set of caveats, running benchmarks is paramount in order to decide which solution will best fit a given workload. Ultimately, using huge pages optimally can be complicated but worthwhile —  we hope this post is a useful starting point. If you find this content interesting and would like to know more about HRT, check out our <a href="https://www.hudsonrivertrading.com/" target="_blank" rel="noreferrer noopener">website</a> or consider applying to <a href="https://www.hudsonrivertrading.com/careers/" target="_blank" rel="noreferrer noopener">join our team</a>!

2022/12/1

Low Latency Optimization: Understanding Huge Pages (Part 1)

Latency is often a crucial factor in algorithmic trading. At HRT, a lot of effort goes into minimizing the latency of our trading stack. Low latency optimizations can be arcane, but fortunately there are a lot of very good <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/8/html-single/optimizing_rhel_8_for_real_time_for_low_latency_operation/index#proc_reducing-cpu-performance-spikes_optimizing-RHEL8-for-real-time-for-low-latency-operation" target="_blank" rel="noreferrer noopener">guides</a> and <a href="https://docs.kernel.org/admin-guide/kernel-per-CPU-kthreads.html" target="_blank" rel="noreferrer noopener">documents</a> to get started. One important aspect that is not often discussed in depth is the role of huge pages and the translation lookaside buffer (TLB). In this series of posts, we’ll explain what they are, why they matter, and how they can be used. We’ll be focusing on the Linux operating system running on the 64-bit x86 hardware, but most points will apply to other architectures as well. We’ve tried to find a balance of providing the most accurate information without digging in the minutiae.  This series of posts is relatively technical, and requires some high-level understanding of operating systems (OS) concepts like memory management, as well as some hardware details such as the CPU caches. In the first post, we will explain the benefits of huge pages. In the <a href="https://www.hudsonrivertrading.com/hrtbeat/low-latency-optimization-part-2/" target="_blank" rel="noreferrer noopener">second post</a>, we will explain how they can be used in a production environment. <h3 class="wp-block-heading">Memory Management 101</h3> The hardware and operating system deal with memory in chunks. These chunks are called pages. For example, when memory is allocated or swapped by the operating system, it’s done in units of pages. On the 64-bit x86 architecture, the default page size is 4 kilobytes (KiB). But x86 64-bit CPUs also support two other page sizes which Linux calls huge pages: 2MiB and 1GiB1. For simplicity’s sake, this series of posts is focused on 2MiB pages. 1GiB pages can also be helpful, but they’re so big that use cases for them tend to be more specialized. 11GiB pages are sometimes called gigantic pages as well <h3 class="wp-block-heading">A quick primer on address translation </h3> When regular programs run, they use virtual addresses to access memory. These addresses are usually just valid within the current process. The hardware and the operating system cooperate to map these into an actual physical address in physical memory (RAM). This translation is done per page (you can probably already see why the size of the page matters). Because programs only see the virtual addresses, on every memory access the hardware must translate the virtual address visible to the program into a physical RAM address (if the virtual address is indeed backed by physical memory). Memory access means any loads or stores from/to the processor for data or instructions regardless of whether they’re cached. These translations are stored by the operating system in a data structure called the <a href="https://en.wikipedia.org/wiki/Page_table" target="_blank" rel="noreferrer noopener">page table</a> which is also understood by the hardware. For each virtual page backed by real memory, an entry in the page tables contains the corresponding physical address. The page tables are usually unique for each process running on the machine. <h3 class="wp-block-heading">Why accessing the page tables might significantly increase latency</h3> Unless the program’s allocator and/or operating system are set up to use huge pages, memory will be backed by 4KiB pages. The page tables on x86 use multiple hierarchical levels. Therefore, looking up the physical address of a 4 KiB page in the page tables requires at least 3 dependent memory loads.2 2 4 loads if <a href="https://en.wikipedia.org/wiki/Intel_5-level_paging" target="_blank" rel="noreferrer noopener">5-level paging</a> is supported by the CPU and enabled in the Linux kernel <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01-1.jpeg" alt="" class="wp-image-273564" style="width:768px;height:497px" referrerpolicy="no-referrer"></figure> The CPU cache will be used to try to fulfill these (similarly to any regular memory access). But let’s imagine that all of these loads are uncached and need to come from memory. Using 70ns as the memory latency, our memory access already has a 70*3=210 nanosecond latency — and we have not even tried to fetch the data yet! <h3 class="wp-block-heading">Enter the translation lookaside buffer</h3> CPU designers are well aware of this problem and have come up with a set of optimizations to reduce the latency of address translation. The specific optimization we are focusing on in this post is the <a href="https://en.wikipedia.org/wiki/Translation_lookaside_buffer" target="_blank" rel="noreferrer noopener">translation lookaside buffer</a> (TLB). The TLB is a hardware cache for address translation information. It contains an up-to-date copy of a number of recently accessed entries in the page tables (ideally all entries in the page tables for the current process). Just like accessing the CPU caches is faster than accessing memory, looking up entries in the TLB is much faster than searching in the page tables3. When the CPU finds the translation it is looking for in the TLB, it’s called a TLB hit. If it does not, it’s a TLB miss. But, just like the regular CPU caches, the TLB size is limited. For many memory-hungry processes, the entire page table’s information will not fit in the TLB. 3 Depending on the state of the cache, it will vary from 3x to ~80x times faster, though some of this latency could be hidden by speculative page walks. <h3 class="wp-block-heading">How big is the TLB?</h3> The TLB structure is not completely straightforward; we’ll approximate it to a set of entries in the hardware. A decent rule of thumb is that recent server x86 CPUs come with a TLB of about 1500-2000 entries per core (<a href="http://www.etallen.com/cpuid.html" target="_blank" rel="noreferrer noopener">cpuid</a> can, for example, be used to display this information for your CPU). Therefore for processes that use 4KiB pages, the TLB can cache translations to cover 2000 (number of entries in the TLB) * 4KiB (the page size) bytes, i.e ~8MiB worth of memory. This is dramatically smaller than the working set of many programs. Even the CPU cache is usually considerably larger! Now, let’s say we are using huge pages instead — suddenly our TLB can contain the translation information for 2000*2MiB = ~4GiB. That is much, much better.   <h3 class="wp-block-heading">Other benefits of huge pages</h3> One huge page covers 512 times more memory than a 4KiB page. That means the number of entries in the page tables is also 512 times smaller for the same working set than if we were using regular pages. This not only greatly reduces the amount of memory used by page tables, but also makes the tables themselves easier to cache. Additionally, the format of page tables for 2MiB is simpler than for regular pages, so a lookup requires one fewer memory access. Therefore, even if a program gets a TLB miss for an address backed by a huge page, the page table lookup will be measurably (or even dramatically) faster than it would have been for a regular page. <h3 class="wp-block-heading">A quick-and-dirty benchmark</h3> No post about optimization would be complete without a completely artificial benchmark 😀. We wrote a simple program that allocates a 32GiB array of double precision numbers. It then adds 130 million random doubles from this array (full source code is available <a href="https://github.com/hudson-trading/hrtbeat/blob/master/huge_memory_bench.cpp" target="_blank" rel="noreferrer noopener">here</a>). On the first run, the program generates a random list of indices in the array then stores them in a file. Subsequent runs will read the file so memory accesses will be the same during each run. We ran this program on an otherwise idle Intel <a href="https://en.wikipedia.org/wiki/Alder_Lake" target="_blank" rel="noreferrer noopener">Alder Lake</a> machine. The baseline time for the initialization part of the program was 40% faster when using huge pages. The array is initialized linearly, which is the best case scenario for the hardware so the speedup is not dramatic. However, when doing random accesses to add the doubles, the runtime is decreased by a factor of 4.5. Note that the number of seconds for a run can vary significantly with small changes in the program or the use of different compilers.  However, the performance improvement for huge pages remains quite clear. <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="615" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1.png" alt="" class="wp-image-273565" style="width:768px;height:461px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1.png 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1-980x589.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02-1-480x288.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> <h3 class="wp-block-heading">When you should not use huge pages</h3> Huge pages should be thought of as an optimization. Just like any other optimization, they may or may not apply to your workload. Benchmarking is important to determine whether it’s worth investing time in setting them up. In the second post of this series, we’ll detail how they can be used and list some substantial caveats. <h3 class="wp-block-heading">Conclusion</h3> On every memory access for code or data, the hardware translates virtual addresses into physical addresses. Using huge pages measurably speeds up this translation. Huge pages also allow the TLB to cover a lot of memory. Ideally, we’d want our entire working set to be translatable by the TLB without ever going to the page tables. This reduces latency of memory access, and also frees some room in the CPU caches (which no longer need to contain as many cached page table entries). If you find this content interesting and would like to know more about HRT, check out our <a href="https://www.hudsonrivertrading.com/" target="_blank" rel="noreferrer noopener">website</a> or consider applying to <a href="https://www.hudsonrivertrading.com/careers/" target="_blank" rel="noreferrer noopener">join our team</a>! Now that we’ve covered the advantages of huge pages, head over to <a href="https://www.hudsonrivertrading.com/hrtbeat/low-latency-optimization-part-2/" target="_blank" rel="noreferrer noopener">part 2</a> to read more about the practicality (and challenges!) of using huge pages in a production environment. <h4 class="wp-block-heading">Further reading</h4> <a href="https://www.akkadia.org/drepper/cpumemory.pdf" target="_blank" rel="noreferrer noopener">What every programmer should know about memory</a>

2022/11/27

Intern Spotlight: 2022 Software Engineering Projects

<h3 class="wp-block-heading" id="h-introduction">Introduction</h3> Summer is intern season at HRT; within a few short months, our interns are truly woven into the fabric of HRT’s culture, adding vibrancy and enthusiasm to our offices around the globe. Working in close collaboration with a supportive network of mentors, teams, and longtime HRTers, interns have the opportunity to tackle some of the most interesting technical problems in the world of automated trading and make a profound and lasting impact on our day-to-day operations. Throughout this past summer, interns and HRTers connected with each other over a jam-packed events calendar, shared projects, and extra-long lunch lines in our cafeteria. This post in our intern spotlight series highlights the work of three of our Software Engineering interns: <ul class="wp-block-list"> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-software-engineering-projects/#h-toucanserver">ToucanServer by Emma Yang</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-software-engineering-projects/#h-cashy-s3-integration">Cashy S3 Integration by Eric Zhang</a></li> <li><a href="https://www.hudsonrivertrading.com/hrtbeat/intern-spotlight-2022-software-engineering-projects/#h-labshell">Labshell by Shahbaz Momi</a></li> </ul> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-toucanserver">ToucanServer</h2> <h3 class="wp-block-heading" id="h-by-emma-yang">By Emma Yang</h3> Over this past summer, I’ve had the amazing opportunity to learn about the engineering problems HRTers are tackling and work on a few projects of my own as a Core intern. One of my projects was ToucanServer, a gRPC and protobuf-based server that expands upon HRT’s manual trading (“click trading”) platforms. <h4 class="wp-block-heading" id="h-the-problem">The Problem</h4> While most of HRT’s trading is wholly automated, the amount of non-automated trading has increased as the company has expanded. To that end, it has become increasingly important to support manual, human-driven trading that allows HRTers to intervene and send orders and cancellations to markets. HRTers currently use Toucan, a command-line interface, for manual trading. However, as more and more teams at HRT need a way to manually enter orders, we increasingly need a more versatile tool that can be adapted to the way different teams at HRT trade. <h4 class="wp-block-heading" id="h-toucanserver-0">ToucanServer</h4> My project was to develop a gRPC server that would expand Toucan into a more flexible platform for manual trading that would expose clients (whether it be semi-automated trading strategies or HRTers opening a custom front-end interface for their team) to the order entry, state management, and position-tracking features of Toucan. <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/01.jpeg" alt="" class="wp-image-273549" style="width:512px;height:413px" referrerpolicy="no-referrer"></figure> Implemented in C++, ToucanServer synchronously handles unary and server-streaming remote procedure calls (RPCs) from connected clients to send orders and cancellations. Clients communicate to ToucanServer by instantiating a gRPC stub — a local object in the client program that allows a client to send requests and receive responses defined in a protobuf specification using methods implemented in the gRPC server. This client-server model makes sending orders and receiving status updates from ToucanServer very intuitive for users building clients for manual trading, as the client can make remote calls to ToucanServer just like calling a local method. <h4 class="wp-block-heading" id="h-multithreading-in-toucanserver">Multithreading in ToucanServer</h4> One of the main technical challenges of building ToucanServer was the use of multithreading. While most of HRT’s infrastructure avoids multithreading due to the overhead of context switching, a service like ToucanServer needs to run a concurrent thread so that the server can listen and respond to client requests. In ToucanServer’s case, since I used a synchronous server model, this thread will block whenever we receive a client request and unblock when the server returns a response. To communicate between the main thread for order entry and the server thread, ToucanServer uses a thread-safe queue of requests for order entry and status responses from clients. Inserting a request into the queue blocks the server thread with a client-specific mutex, waiting until the request has been processed by the order entry thread and passing the result back to the client before processing any additional requests from that client. <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/02.jpeg" alt="" class="wp-image-273550" style="width:768px;height:295px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">The server handles a client request by queuing it in the thread-safe queue, which is picked up by the order entry thread</figcaption></figure> <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/03.jpeg" alt="" class="wp-image-273552" style="width:768px;height:295px" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">The order entry thread builds a response, which it places into the thread-safe queue for the server to retrieve and return to the client</figcaption></figure> Not only does this multithreaded design enable us to run a synchronous server concurrently with the order entry thread, it also enables us to expand Toucan from a single-client-per-process tool to a single process that can handle multiple clients at once. Since gRPC automatically multiplexes client requests across its own thread pool, ToucanServer can handle both concurrent clients and concurrent client requests. <h4 class="wp-block-heading" id="h-impact">Impact</h4> ToucanServer’s main purpose is to enable trading teams to build web apps and other front-end interfaces that connect to ToucanServer, send instructions through RPCs, and display the information contained in status updates that are streamed to the client. Each team will have the flexibility to build an interface based on the view of order state that is most useful to them – a significant unlock for our manual trading platforms. <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-cashy-s3-integration">Cashy S3 Integration</h2> <h3 class="wp-block-heading" id="h-by-eric-zhang">By Eric Zhang</h3> For high-performance applications, data needs to be close to where it’s processed. HRT achieves this through Cashy — a distributed cache that enables rapid data retrieval with high scalability for compute cluster applications. Cashy utilizes a hierarchy that ranges from local memory to remote solid state drives, with many layers in between, and acts as a proxy client to a distributed file system for reads at significant scale. To make a request, applications make a remote procedure call to a Cashy daemon on the same machine. That local Cashy process turns a single request into many block requests, which it sends to many other caching hosts in parallel. Finally, Cashy returns the response over shared memory, allowing it to rapidly service requests. Cashy’s default configuration is typically used for servicing NFS read requests, however, many applications use object stores like Amazon’s Simple Storage Service (S3). In this project, we extend Cashy to serve requests in the S3 format and read from S3 stores. This allows applications that use S3 to be able to run at scale in our compute clusters. <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="465" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04.png" alt="" class="wp-image-273553" style="width:512px;height:233px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04.png 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-980x445.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/04-480x218.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">Orange components are the proposed additions to Cashy</figcaption></figure> <h4 class="wp-block-heading" id="h-s3-server">S3 Server</h4> One simple way to serve S3 requests is to set up HTTP servers that are Cashy clients and proxy the requests from S3 clients to Cashy. However, this creates performance overhead due to extra interprocess communication and requires additional deployment and monitoring. Instead, we implement a subset of S3 functionality in Cashy itself. Since the Amazon S3 API is large, we start by only implementing the GetObject action, which retrieves S3 objects. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<span role="button" tabindex="0" data-code="GET /file.txt HTTP/1.1 Host: somebucket.somehost.com Date: Mon, 3 Oct 2022 22:32:00 GMT Authorization: authorization string" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code>GET /file.txt HTTP/1.1 Host: somebucket.somehost.com Date: Mon, 3 Oct 2022 22:32:00 GMT Authorization: authorization string</code></pre></div> Amazon S3 GetObject request All other actions can be handled by a slower, but fully-featured S3 implementation without much performance loss. Since Cashy operates in an asynchronous model, all networking is done using non-blocking sockets controlled by an epoll-based event loop. To perform a GetObject, a Cashy worker process begins by receiving an HTTP S3 request. This S3 request is then translated by an S3 parser into a Cashy request. Once Cashy receives the request, it translates the object read request into multiple block requests. As an optimization, Cashy prefetches blocks so that they will be preloaded into faster tiers of storage hopefully before the client asks for them. One tricky corner case is that for many S3 reads, the file size is unknown. Therefore, we always prefetch at most the number of blocks that have already been read, to avoid prefetching more than twice the number of blocks necessary. Earlier we mentioned that S3 actions other than GetObject will be handled by a different, fully-featured S3 implementation. Therefore, the Cashy S3 server needs to redirect requests it can’t handle. Despite what the <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingRouting.html">Amazon S3 documentation</a> states, S3 clients like Boto3 <a href="https://github.com/boto/boto3/issues/2708">do not properly support HTTP redirects</a>. To work around this, we built functionality for the Cashy S3 server to proxy S3 requests. Here, we had to be careful to re-authenticate the requests because the host has changed (see <a href="https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html">Amazon Signature V4</a>). <h4 class="wp-block-heading" id="h-s3-reader">S3 Reader</h4> Now that applications can use the S3 protocol to read from Cashy, we’d also like Cashy to read from S3. Our S3 reader needs to support the following interface: <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<span role="button" tabindex="0" data-code=" // IReadCashyBlocks impl --------------------------------------------------- void beginReadBlock(BlockReadInfoPtr blockInfo) override; void cancelReadBlock(const BlockReadInfo& blockInfo) override; using CashyReadBlockCallback = Callback<void(BlockReadInfoPtr blockInfo)>; ---------------------------------------------------" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code> // IReadCashyBlocks impl --------------------------------------------------- void beginReadBlock(BlockReadInfoPtr blockInfo) override; void cancelReadBlock(const BlockReadInfo& blockInfo) override; using CashyReadBlockCallback = Callback<void(BlockReadInfoPtr blockInfo)>; ---------------------------------------------------</code></pre></div> A CashyReadBlockCallback is expected for each beginReadBlock that has not been canceled. To implement this efficiently, each Cashy worker has a connection pool for each S3 host that Cashy can retrieve information from. Requests wait in queue until a connection to the corresponding host becomes available. Then, we send the request, wait for a response, and perform the read callback. All of this is done asynchronously to avoid blocking. Overall, the state machine roughly looks like: <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="829" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05.webp" alt="" class="wp-image-273554" style="width:512px;height:415px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05.webp 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05-980x793.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/05-480x389.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"><figcaption class="wp-element-caption">Simplified model of S3 Reader. Solid lines represent direct transitions and dotted lines represent asynchronous transitions managed by an <a href="https://man7.org/linux/man-pages/man7/epoll.7.html">epoll</a>-based event loop</figcaption></figure> <h4 class="wp-block-heading" id="h-conclusion">Conclusion</h4> Before, Cashy could only support the following flow: <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="81" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06.webp" alt="" class="wp-image-273555" style="width:512px;height:41px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06.webp 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-980x78.webp 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/06-480x38.webp 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> Now Cashy supports clients using S3 to read from Cashy’s underlying filesystem. Moreover, existing Cashy clients can now read from S3 stores. Given that Cashy can both serve and read from S3, we can also now use Cashy as a high-performance cache for S3 services! <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="474" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07.png" alt="" class="wp-image-273556" style="width:512px;height:237px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07.png 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07-980x454.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/07-480x222.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"> <div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div> <h2 class="wp-block-heading" id="h-labshell">Labshell</h2> <h3 class="wp-block-heading" id="h-by-shahbaz-momi">By Shahbaz Momi</h3> Imagine that you want to set up a live trading environment that relies upon dozens of configuration files and just as many, if not more, binaries running across several hosts. Now imagine you want to interact with this environment, swapping out these binaries in real-time and changing configuration on the fly. How would you approach this problem? HRT has rolled out its own testing framework called Labtest which attempts to solve this issue, but it has some shortcomings; namely, it doesn’t have the interactivity desired as it only supports automated, closed-loop testing. My first project over the course of my summer internship at HRT was to develop Labshell, an interactive shell that aims to fill in this gap.  <h4 class="wp-block-heading" id="h-architecture">Architecture</h4> Labtest contains lots of features around deployment and configuration. However, it doesn’t provide any means of interactivity; it is a run-once style of program, and this comes with a few challenges, specifically around architecture. While minor changes to the environment can be made (e.g. restarting a process), major changes such as restarting processes on different hosts require tearing down the whole environment. To account for this, I chose to use an immutable data flow architecture with a central render loop passing data in one direction, allowing for effective handling of both types of changes. <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="482" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/08.png" alt="" class="wp-image-273557" style="width:768px;height:362px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/08.png 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/08-980x461.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/08-480x226.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> Minor changes can be performed directly, while major changes are collated and executed in batch by the Choreographer at the end of the cycle. Immutability is important here so that the currently running environment can’t be modified, but rather has to be evolved for a major change. <h4 class="wp-block-heading" id="h-concurrency">Concurrency</h4> To achieve interactivity, we need a separate UI thread which is never blocked and ready to receive user input. In isolation, this is easily solved by simply creating a new background thread. However, we also have to communicate with Labtest threads running background jobs (e.g. sending keepalives to remote hosts, running processes, etc.) while managing cancellations and exceptions that occur in these jobs. The solution that Labtest internally employs is to use Trio, a Python library for async concurrency. With Trio you can create tasks which are essentially coroutines running on a single thread (to get around GIL limitations). We can leverage coroutines to have the small, periodic tasks that Labtest needs to schedule run on the same thread as the UI thread, avoiding any concurrent read and write issues. <figure class="wp-block-image aligncenter size-full is-resized"><img decoding="async" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/09.jpeg" alt="" class="wp-image-273558" style="width:768px;height:176px" referrerpolicy="no-referrer"></figure> Note that certain tasks, like creating a new process, actually create new threads internally — trio manages this for us by creating thread-safe communication channels and managing the thread’s lifetime. <h4 class="wp-block-heading" id="h-user-interface">User Interface</h4> The user interface is perhaps the most important part of this project; how do we develop something efficient that people actually want to use? Throughout the design and creation of the UI, this was facilitated by a number of key features. The UI framework I used to render the shell was prompt_toolkit, an excellent library which made it easy to facilitate these features while also being aware of the hidden challenges mentioned above. <h4 class="wp-block-heading" id="h-autocompletion">Autocompletion</h4> Autocompletion is probably the largest time saver, but with it comes a number of other questions, such as how to generate these completions, how to order the completions, and how do they evolve as the user types in additional input? <figure class="wp-block-image aligncenter size-full"><img loading="lazy" decoding="async" width="512" height="145" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/10.gif" alt="" class="wp-image-273559" referrerpolicy="no-referrer"></figure> <h4 class="wp-block-heading" id="h-inferencing">Inferencing</h4> If a user doesn’t have to type input, why should they? This is another huge time saver, as this means the user can omit many parameters to commands. <div class="wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers" data-code-block-pro-font-family="Code-Pro-JetBrains-Mono" style="font-size:clamp(16px, 1rem, 24px);font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#babed8;--cbp-line-number-width:calc(1 * 0.6 * 1rem);line-height:clamp(24px, 1.5rem, 36px);--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)">C++<span role="button" tabindex="0" data-code="# A command which accepts 2 parameters: command param_1 param_2 # param1 = ["option_1"], param2 = ["option_2", "option3"] # All of these are valid since we can infer param_1: command option_2 command option_1 option_3 # Invalid since we can't infer param2: command command option_1" style="color:#babed8;display:none" aria-label="Copy" class="code-block-pro-copy-button"><svg xmlns="http://www.w3.org/2000/svg" style="width:24px;height:24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2"><path class="with-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4"></path><path class="without-check" stroke-linecap="round" stroke-linejoin="round" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"></path></svg><pre class="shiki material-theme-palenight" style="background-color: #292D3E" tabindex="0"><code># A command which accepts 2 parameters: command param_1 param_2 # param1 = ["option_1"], param2 = ["option_2", "option3"] # All of these are valid since we can infer param_1: command option_2 command option_1 option_3 # Invalid since we can't infer param2: command command option_1</code></pre></div> But it also comes with its own considerations like identifying sensible defaults, and addressing how to handle cases where multiple inferences are possible. <h4 class="wp-block-heading" id="h-conclusion-0">Conclusion</h4> In the end, I had a shell which was able to start bash and SSH sessions into remote hosts, reconfigure the live environment, restart and debug running processes, and launch our internal tooling preconfigured to the simulated trading environment. <figure class="wp-block-image aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="1024" height="713" src="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/11.png" alt="" class="wp-image-273560" style="width:512px;height:357px" srcset="https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/11.png 1024w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/11-980x682.png 980w, https://www.hudsonrivertrading.com/wp-content/uploads/2024/04/11-480x334.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw" referrerpolicy="no-referrer"></figure> <h2 class="wp-block-heading" id="h-like-what-you-see">Like what you see?</h2> If you find the featured projects exciting, consider <a href="https://www.hudsonrivertrading.com/careers/">applying</a>!

2022/11/9