Today's Reads

Hard drives are still the default choice for many homelab and data hoarding enthusiasts. They still hold the gigabytes per dollar advantage over SSD-s (for now), and if you buy the big external drives and take the drives out of the enclosures, you can get a pretty good deal.

Spinning rust has one obvious downside: it’s slow, both in maximum transfer speeds and in latency. For most use cases this is fine, but if you run a service that depends on a database or if you have a lot of clients relying on that storage, you’ll find that hard drives just won’t cut it.

At this point you probably don’t want to splurge and build an all-SSD NAS, so you look up caching solutions. Depending on your platform, you’ll have all sorts of options.

If you run ZFS, you’ll likely learn about L2ARC and why you probably don’t want to use it (hint: check your ZFS ARC hit/miss ratio first). Or SLOG, or the special metadata device. And in most cases the top recommendation is to add as much RAM as you can to the system so that you can benefit from a bigger filesystem cache. You’ll see some improvements, but not all workloads will benefit from this and you might up being disappointed after all that work.

I cannot speak for options on other platforms, but in most cases the idea is similar: buy a separate sacrificial SSD and use it as a cache drive.

Cache is (not) king

Take a moment and think about what type of data you’re storing.

Here’s an overview based on my own setup:

• archived YouTube videos (bunch of big files)
• cat pictures (bunch of small and big files)
• services that utilize a database (bunch of smaller files)
• various installation media (bunch of big files)
• backups of physical disks (bunch of big files)
• a copy of my Steam library (bunch of big and small files)
• a web server (bunch of small files)

Most of the data I have is accessed relatively infrequently, and when it is, the performance requirements are not that high. Any hard drive based array will be able to handle streaming video or copying big bulky files over the network.

Data that is more latency sensitive, such as Nextcloud, Jellyfin (and its .sqlite DB) and PostgreSQL, take up a relatively small part of the overall storage.

In this case (and assuming that the setup allows for it physically), the solution is simple: add a smaller flash-based storage pool to the setup and use it for data that benefits from it.

I experimented with a similar setup in the past with two 8 TB hard drives holding all the big files and a pair of 250 GB SSD-s handling Nextcloud and other services. Now I had the best of both worlds.

In short, you don’t need to mess around with a fancy caching solution if your storage usage patterns are simple and predictable. If you do a little bit of investigation once, you can bypass all the logic that is usually built into various caching solutions and get a better end result as no caching solution is 100% perfect.

This is pruned from Engineering strategy.

When you’re writing an engineering strategy, what you put in is important, but it’s almost as important to be deliberate about what you choose to leave out. The full list of things to exclude is uncountably vast, but there are a few worth emphasizing in particular.

LinkedIn’s Jeff Weiner documented their hierarchy of operating documents as Vision, Mission, Strategy, Objectives, Priorities, Culture, and Values (which some companies have standardized as VSMO). Some of those are very helpful, but two to leave out when writing an engineering strategy are “culture” and “values.” These are quite important, but must be handled at the company scope rather than the engineering scope. Organizations within a company that go too far in shaping values distinct from the company values inevitably create a Values Oasis: a short-term workaround that requires more and more maintenance energy from your organizational leadership until eventually they have little time for anything else. If you want to define engineering specific values, channel that energy towards aligning with the broader company’s values instead.

A third VSMO idea that I’ve excluded is “mission.” I’ve almost universally found that team and organizational mission statements create more confusion and misalignment than they resolve. Mission statements are an attempt to rewrite reality with force of desire, which is, I’m sad to say, folly. Mission statements are treated as statements of truth about what a team ought to be doing, but this is a fundamentally misunderstanding of why teams exist: they exist to support the business, and in the case of engineering teams, they generally do so by supporting other teams. A team’s purpose depends much more heavily on what their users need than on what they want to be working on.

Further, mission statements fall afoul of both the “don’t generalize” and “don’t lie” rules of effective engineering strategy. Most mission statements create confusion by articulating the team’s desired state rather than their actual state. They also tend to be overly broad in a way that causes incompatible interpretations. Finally, they tend to overstep to “claim territory” from other related teams, wasting energy on folks competing to “own” areas that neither team is capable of actually supporting. Rather than a team mission, write a team strategy explaining your guiding policies to navigate your current circumstances.

Finally, abandon your plan to proclaim axiomatic truths. It’s extremely tempting to start out your engineering strategy by writing a series of bold statements about reality—Simple code is good code! Optimize for discounted future developer velocity!—but lists of commandments are dead documents on arrival that cannot evolve with your organization as you grow. Instead of writing axioms, write a strategy explaining why these are important and fold it into your quality measure. That will create a living document that folks can evolve with you and lives beyond your tenure.

January 1st, 2016 ~ by admin

For an n by n real matrix A, Hadamard’s upper bound on determinant is

where a_ij is the element in row i and column j. See, for example, [1].

How tight is this upper bound? To find out, let’s write a little Python code to generate random matrices and compare their determinants to Hadamard’s bounds. We’ll take the square root of both sides of Hadamard’s inequality to get an upper bound on the absolute value of the determinant.

Hadamard’s inequality is homogeneous: multiplying the matrix A by λ multiplies both sides by λⁿ. We’ll look at the ratio of Hadamard’s bound to the exact determinant. This has the same effect as generating matrices to have a fixed determinant value, such as 1.

    
    from scipy.stats import norm
    from scipy.linalg import det
    import matplotlib.pyplot as plt
    import numpy as np
    
    # Hadamard's upper bound on determinant squared
    def hadamard(A):
        return np.prod(np.sum(A**2, axis=1))
                
    N = 1000
    ratios = np.empty(N)
    
    dim = 3
    for i in range(N):
        A = norm.rvs(size=(dim, dim))
        ratios[i] = hadamard(A)**0.5/abs(det(A))
    
    plt.hist(ratios, bins=int(N**0.5))
    plt.show()
    

In this simulation the ratio is very often around 25 or less, but occasionally much larger, 730 in this example.

It makes sense that the ratio could be large; in theory the ratio could be infinite because the determinant could be zero. The error is frequently much smaller than the histogram might imply since a lot of small values are binned together.

I modified the code above to print quantiles and ran it again.

    print(min(ratios), max(ratios))
    qs = [0.05, 0.25, 0.5, 0.75, 0.95]
    print( [np.quantile(ratios, q) for q in qs] )

This printed

    1.0022 1624.9836
    [1.1558, 1.6450, 2.6048, 5.7189, 32.49279]

So while the maximum ratio was 1624, the ratio was less than 2.6048 half the time, and less than 5.7189 three quarters of the time.

Hadamard’s upper bound can be very inaccurate; there’s no limit on the relative error, though you could bound the absolute error in terms of the norm of the matrix. However, very often the relative error is moderately small.

More posts on determinants

• Relatively prime determinants
• Accurately computing a small determinant
• Tridiagonal systems

[1] Courant and Hilbert, Methods of Mathematical Physics, Volume 1.