• Dev Notes
  • Posts
  • Uber Migrates 1 Trillion Records to Save $6 Million Annually

Uber Migrates 1 Trillion Records to Save $6 Million Annually

Good morning! Uber has migrated a trillion records to a custom data store called LedgerStore, saving $6 million annually. Microsoft and Coursera have launched four new entry-level tech certificates to help people break into IT, cloud, business analyst, and project management roles. The popular malware analysis tool YARA has been rewritten from the ground up in Rust, called YARA-X, offering improved performance, reliability, and developer-friendly APIs.

Uber Migrates 1 Trillion Records to Save $6 Million Annually

Uber was dealing with crazy storage costs from using DynamoDB and blob storage for their massive load of payment transaction data. So they got clever and built their own custom data store called LedgerStore, designed to guarantee solid data integrity for financial transactions while saving them an insane $6 million per year.

LedgerStore has amazing technology under the hood:

  • A two-phase commit process for strongly consistent indexes

  • Materialized views pulled from Uber's Docstore database for eventually consistent indexes

  • The ability to automatically reindex data when definitions change

But migrating a trillion records (we're talking petabytes of data here) from the old systems to LedgerStore was not easy. Uber's team got creative:

  • Shadow validation by double writing to DynamoDB and LedgerStore to cross-check reads

  • Offline checks on historical data, coupled with an Apache Spark backfill job

  • Carefully controlling the rollout, rate-limiting the backfill to 10x typical load with an emergency stop option

  • Initially falling back to fetching data from DynamoDB if not in LedgerStore yet

All that effort paid off big time. They nailed the migration seamlessly, with no downtime. And the payoff? A streamlined storage setup tailored to their massive scale, while raking in millions in savings.

Read More Here

Microsoft Adds Four New Entry-Level Certificates to Coursera

Microsoft and online course platform Coursera are teaming up to launch some fresh tech training programs aimed at helping people break into entry-level roles, no prior experience required.

They're rolling out four new Professional Certificate courses:

  • IT Support Specialist - This one covers the crucial skills for IT support jobs like data backup, cloud computing fundamentals, and managing mobile devices. It's available now.

  • Cloud Support Associate - Focused on Microsoft's Azure cloud services and solutions. You'll learn how to manage cloud resources, security, cost optimization for Azure - everything you need to get started in cloud support. Also available now.

The other two new certs are:

  • Business Analyst - The first business analyst program on Coursera. It combines business smarts with tech and data science know-how. You'll pick up strategic decision making, change management, process modeling, and more sweet skills.

  • Project Manager - With project management roles on the rise, this cert is all about mastering PM principles, methodologies, tools...especially Microsoft Project for planning and scheduling projects. It launches this summer but you can pre-enroll now.

A few key things to note about these new programs:

  • Hands-on projects so you get practical experience

  • They'll help prepare you for official Microsoft certification exams

  • Build on Microsoft's existing lineup of beginner Coursera certs (over 600k enrolled already)

  • Use a conversational teaching style to make the material engaging

The bigger vision here, according to Microsoft's Elisa Graceffo, is to "provide accessible upskilling paths that fill in-demand skill gaps" regardless of people's backgrounds. It's about opening more doorways into tech careers.

Read More Here

YARA is dead, long live YARA-X

For over 15 years, YARA has been the trusty pattern-matching tool in every malware researcher's toolkit. But now, the folks at VirusTotal have some big news - they've announced YARA-X, a total rewrite of YARA built from the ground up in Rust.

So what's the deal with this new YARA-X? Well, a few key things:

  • Better user experience - we're talking a slick new command-line interface, clearer error reporting, the works. They want to make this thing a joy to use.

  • Damn near 100% compatibility with your existing YARA rules. No need to start from scratch.

  • Significantly faster performance, especially for complex rules involving regular expressions or loops that tend to bog YARA down.

  • Enhanced reliability and security thanks to being built with Rust's memory safety guarantees.

  • Developer-friendly with official APIs for Python, Go, and C to easily integrate YARA-X into other tools.

YARA-X has already been tested at VirusTotal, scanning millions of files with thousands of rules. Even in beta, it's insane.

The original YARA isn't going anywhere just yet. VirusTotal will still maintain it with bug fixes and minor updates for the faithful users.

Read More Here

🔥 More Notes

Youtube Spotlight

How AI tells Israel who to bomb

Was this forwarded to you? Sign Up Here