How Netris Enabled
Databento’s infrastructure for hypergrowth
- Executive Summary
- About Databento
- Building Private Cloud Infrastructure across Three Colocation Data Centers
- Standing up Databento’s Private Cloud
- Evaluating the Existing Technology Stack
- Netris to the Rescue
- Reference Architecture
- Initial Result
- Improving Engineer productivity
- Using the Rest API and IaC (Infrastructure as Code) for Automation
- Long Term Results
Netris empowered our early engineering team to launch and operate our own private cloud infrastructure. We have expanded to three data centers in less than a year, keeping up with the exponential growth of our customer base.
Listen to Netris Connect Podcast with Databento’s engineers Rick Fowler and Craig Smith:
Databento’s infrastructure team was tasked to design, build, and further operate its own private cloud infrastructure, spun across three data center regions, and set to grow fast to keep up with the exponential growth of its business. Because the team wanted to make the infrastructure easy to consume and scale, much like the public cloud but on their own hardware, they chose to use Netris Automatic VPC Networking software instead of building in-house network automation from scratch. Databento’s network engineers provided the Netris console to their DevOps and InfraOps engineers for self-serviceability. The high-level abstraction of Netris VPC came across as intuitive for most engineers, both for cloud practitioners and traditional data center operators. DevOps and InfraOps could easily provision (load balancers, network access lists, virtual networks, etc.). Network Engineers could focus on strategic tasks versus wasting time on daily change requests.
The results were a dramatic (10X) increase in engineering teams’ productivity:
- The team eliminated the time DevOps/InfraOps needed to wait for request fulfillment.
- Reduced the time network engineers needed to spend on repetitive tasks.
- Eliminated the need for Network Engineers to engage in writing and troubleshooting low-level procedural automation scripts.
- Databento used Linux and a robust open-source software ecosystem combined with performant modern commodity hardware. Similar to what major cloud providers are building their infrastructures on but without operational overhead, as Netris software takes care of control plane configuration and overall network management lifecycle.
Databento makes it simpler and faster to access institutional-grade financial data. Founded by traders and engineers from among the world’s leading quantitative hedge funds, Databento’s self-service model allows users to instantly pick up live exchange feeds and terabytes of historical data – and only pay for what they use. Databento’s servers are hosted in the colocation facilities of various trading venues for low-latency and high-fidelity data capture, direct from the source.
To learn more about Databento, visit www.databento.com.
Netris Connect Interview
Christina Qi, CEO, Databento
Building Private Cloud Infrastructure across Three Colocation Data Centers.
Rick Fowler and Craig Smith are Systems Engineers at Databento. They work on the Infrastructure team, whose mission is to ensure stable operations of the network, compute, and storage systems, as well as to ensure DevOps and InfraOps engineers can elastically spin up network services such (i.e., load balancers, V-Nets, NAT, DHCP, VPN, etc.)
Standing up Databento’s Private Cloud
During Databento’s early seed stage days, Luca Lin, the founding CTO, had originally put together a simple two-track infrastructure to get Databento off the ground.
As business picked up and started growing fast, Rick Fowler was hired by Luca Lin further to redesign and stand up their version 2 infrastructure with growth, agility, and scalability in mind. They wanted to achieve cloud-like operational efficiency but run their own infrastructure in colocation data centers.
We needed EVPN/VXLAN as a scalable foundation for the network. We chose Nvidia Spectrum switches for the best quality/price/performance ratio. We also required Firewalling, Routing, Load Balancing, VPN, NAT, and DHCP. Netris VPC ideally matched our requirements for the comprehensive operation of all aspects of our network.
Evaluating the Existing Technology Stack
Rick first examined the existing network toolset to see if it would solve his problem. That initial setup was based on traditional enterprise data center architecture with manual CLI configuration of switches and routers, storing IP address subnet and configuration info in spreadsheets, and it lacked any automation. It was a simple solution for the early days, never meant for scalability.
Rick is savvy with Ansible, so he initially considered buying Cisco/Juniper routers and firewalls, F5 for load balancing, and some switches and leveraging Ansible for automation. But this solution had three significant downsides for Databento. 1) These traditional networking vendors provide very little abstraction. Even with the most advanced Ansible skills, Rick must design imperative procedural automation, maintain it across multiple vendors, and apply individual logic for each network functionality. It’s very time-consuming and prone to human error. 2) DevOps and InfraOps engineers would not get any self-serviceability; every request would need to go through Rick and Craig, the Network Engineering team introducing delays and risk for configuration drift. 3) Hard to ask for vendor help when multiple products are involved — Lack of centralized support (single throat to choke).
Rick wanted, and Luca required, a general and elegant solution to let them quickly and easily manage the infrastructure and provide self-serviceability to the entire engineering team. They needed to make infrastructure robust and highly scalable – but preserve the startup-spirited agility.
Netris to the Rescue
One of Databento’s investors is Mike Dvorkin, a well-known figure in the network infrastructure space commonly referred to as “the creator of Cisco ACI,” and he suggested Rick check out Netris.
Netris allows engineers to design, build, and operate fully automated network infrastructures providing the entire engineering team with cloud-like VPC networking abstraction via the intuitive web console, API, and IaC (Infrastructure as Code). Netris leverages SmartNIC accelerated Linux networking running over conventional servers for (Layer 3-7) VPC networking services (Routing, firewalling, load balancing, NAT, DHCP, VPN, etc.) For physical (Layer 1/2) switching, Netris can automate the lifecycle of open-networking switches (Nvidia, Dell, Edge-Core) with Linux OS (Cumulus, SONiC, Ubuntu/SwitchDev), can automate API-driven bare-metal platforms (like Equinix, PhoenixNAP, etc.), or can run anywhere over ethernet switches statically pre-configured with a carved-out range of VLANs.
Before investing in hardware, how do you evaluate a solution of this scale (3 regions, six border routers, 30+ switches)?
Rick has used Netris’s on-demand sandbox to learn and evaluate the platform. He reviewed pre-configured examples and walked through step-by-step “learning by doing” guide. Netris Sandbox quickly allowed him to simulate his desired architecture, create the anticipated services, and perform end-to-end testing with realistic end-user traffic flows. Unlike Cisco/Juniper/F5 sales motion, there’s no need to wait for loaner equipment – Netris Sandbox is available immediately at https://netris.io/sandbox.
For switches (Layer 1-2), Rick chose Nvidia Spectrum switches (formerly Mellanox) with 100Gbps ports and Cumulus Linux NOS because he worked with Nvidia/Mellanox switches before at Canonical (the maintainer of Ubuntu Linux) and was impressed with the hardware quality and low switching latency.
Instead of buying Cisco/Juniper routers, firewalls, and F5, Rick chose Netris SoftGate PRO, the 100Gbps version of the VPC gateway combined with the Nvidia ConnectX-6 SmartNIC card running on a Dell server with dual Intel Xeon CPUs.
Rick says, “I could use Ansible to scratch-build in-house automation and undertake the ongoing maintenance of my automation pipeline, which would take much of my time. Also, my DevOps/InfraOps colleagues may not be able to self-service. Netris came with my desired automation already built-in and with cloud-like VPC Networking abstraction that our entire engineering team found intuitive. When I need automation specific to Databento’s business, I can leverage Netris API through config management like Ansible, Salt, or Terraform, or use Python.”
Netris subscription includes 24/7/365 support which is responsible for the end-to-end operation of the entire system, not just Netris software. Even if there is a hardware issue Netris support team can help to narrow that down and make it easier for you to work it out with your hardware supplier, or if there is an issue with one of the open-source packages, Netris will take care of that too.
Rick says, “Working with Netris TAC over slack feels like working with colleagues on my team.”
Netris Connect Interview
Rick Fowler & Craig Smith,
The use case for Netris SoftGates (VPC gateways) is particularly beneficial. We use SoftGates for our web-facing Internet, server Load Balancing, Firewalling, NAT, and DHCP. One-stop-shopping with SoftGate prevented us from having dedicated firewalls, routers, and load balancers. We have these features on SoftGate that support a high number of packets, sessions, and bandwidth utilizing a regular server with a SmartNIC card. And the fact that it is managed through a single Netris controller is particularly of a benefit.
Network Hardware and Software
Network Services (Layers: 3-7)
VPC Gateway HW: Dell server, 2x Intel® Xeon® Gold 5315Y Processor, 128GB RAM
VPC Gateway NIC: Nvidia ConnectX-6 SmartNIC 2x 100GbE
OS: Ubuntu Linux
Control plane: Netris-managed FRR + other open-source Linux networking software
VPN: Netris-managed Wireguard
Elastic Load Balancer: Netris
Router Data Plane: Netris
NAT & Firewall Data Plane: Netris
DHCP: Netris-managed ISC – Kea
Ethernet Switch System (Layer 2)
Spine Switches: Nvidia Spectrum MSN3700C 32x QSFP28 100GbE
Leaf Switches: Nvidia Spectrum MSN2410 48x SFP28 25GbE + 8x QSFP28 100GbE
NOS: Nvidia Cumulus Linux
Switch Fabric Management: Netris
Using Netris, with its built-in automation and VPC Networking abstraction, Rick Fowler quickly deployed the main region in NYC. Craig Smith, an experienced Network engineer, has joined Databento and started working with Rick. Although Craig was new to Netris, he found it intuitive and efficient. Shortly Craig and Rick launched their Boston and Chicago data centers. What commonly takes 6-9 months of development of home-grown automation – Rick and Craig achieved in about one week, which provided them with enough time to take care of everything else. Databento’s private cloud launch was a major success. Databento got the flexibility of the cloud but for its own infrastructure – maintained independence from hardware vendors and cloud providers.
Improving Engineer productivity with Self-Service Private Cloud Infrastructure
Rick and Craig worked with traditional (Cisco/Juniper) networking at previous jobs and used Ansible for automation. Unlike the public cloud, the regular Cisco/Juniper model lacks abstraction, so only network engineers can implement network changes. With that older model, DevOps and InfraOps engineers must create a network change request and, on average, wait 3-10 days to get network engineers to implement the request.
Netris is designed to amplify network engineers’ productivity as well as enable them to offer safe self-serviceability to the entire engineering team.
Rick and Craig have granted access to the Netris web console to their DevOps and InfraOps engineers, who are experienced in Linux and systems engineering; although they are not network engineers (they never configured any Cisco switch or router), they were able to self-service through Netris console.
Daily network change implementation productivity has improved by over 90% from days to minutes.
Using the Rest API and IaC (Infrastructure as Code) for Automation
Netris exposes its VPC-like abstraction also over Rest API and IaC (Infrastructure as Code) to allow engineers to automate network infrastructure programmatically. Most network vendors offer little to no abstraction; hence their programmatic interfaces require engineers to handle many implementation details; even when they support Terraform, it’s not as declarative as cloud practitioners would expect; instead, it is simply a mapping of imperative commands to HCL (Hashicorp Configuration Language). Versus Netris, like VPC in the public cloud, provides a high level of abstraction that makes declarative code possible.
Declarative code is like a thermostat: you set the target temperature, and the system automatically kicks on or off to continuously meet the desired state. The thermostat abstracts away temperature sensors and control relays, so the user can think in terms of outcomes (e.g., the desired temperature) instead of wasting time on implementation details.
Most engineers working for Databento had prior automation experience, including at AWS and GCP, so both Network Engineers and DevOps/InfraOps incorporated automation into their deployment process.
Long Term Results
From a Network Engineering perspective, this solution allowed Databento to deploy and operate efficiently multi-region and scalable data center networks quickly and safely. Have holistic software to manage all aspects of the network. Stay hardware agnostic and not afraid of current or future supply chain issues.
From DevOps and InfraOps perspectives, they benefit from the self-service capability of Databento’s network, consuming it with the operational efficiency of the public cloud.
Here’s a summary of how Netris helped Databento to build and operate its own multi-region private cloud infrastructure network.
- Amplified Network Engineers’ productivity through automatic network operations and declarative automation API
- Derisked Databento’s growth from supply chain limiting factors by enabling hardware vendor independence.
- Boosted engineering team’s productivity through self-service VPC network model and Improved collaboration between DevOps, InfraOps, and Network Engineering teams