DEV Community: Silkline

Turning a photo into a cross stitch chart: picking threads with CIEDE2000

Silkline — Mon, 02 Nov 2026 06:03:34 +0000

A cross stitch chart looks like a pixel art problem. Shrink the photo, reduce the palette, done. Then you stitch it, and the sky has a stripe of grey-green nobody asked for.

I built a browser tool that turns photos into charts, and most of the interesting work was in one question: which threads should this picture use? Unlike a normal palette, you can't invent colours. DMC stranded cotton has 489 of them, and every colour you choose is another skein to buy and another thread change.

Why RGB distance gets it wrong

The obvious way to match a pixel to a thread is Euclidean distance in RGB. It's cheap and it's wrong in a visible way, because equal steps in RGB aren't equal steps to the eye. Two dark greens can be numerically far apart and look the same, while two pale skin tones can be numerically close and look different.

So every colour goes through the same pipeline first:

sRGB bytes to linear light (undo the gamma curve).
Linear RGB to CIELAB, with a D65 white point.
Compare in Lab with CIEDE2000.

CIEDE2000 is long, with corrections for lightness, chroma and the awkward blue region, so I wouldn't write it from memory. The paper by Sharma, Wu and Dalal (2005) publishes test pairs, and those belong in your unit tests.

Speed matters too. A large chart can have hundreds of thousands of pixels, so the hot paths write Lab values into a typed array rather than allocating an object per pixel:

export function rgbToLabInto(r: number, g: number, b: number, out: Float32Array | Float64Array, o: number): void {
  linearToLabInto(SRGB_TO_LINEAR[r], SRGB_TO_LINEAR[g], SRGB_TO_LINEAR[b], out, o);
}

SRGB_TO_LINEAR is a 256-entry lookup table, so the gamma step costs one array read.

Choosing the threads: greedy, from a fixed list

Quantisation methods such as k-means produce colours that don't exist as thread. You can snap them to DMC afterwards, but two clusters may land on the same skein, and you end up with fewer colours than you asked for.

Working from the fixed list avoids that. The tool adds one thread at a time, always the one that brings the pattern closest to the photo, until it reaches the requested count. Simplified, the idea is:

// Simplified sketch, not the production loop.
function pickThreads(pixels: Lab[], threads: Thread[], count: number): Thread[] {
  const chosen: Thread[] = [];
  while (chosen.length < count) {
    let best = threads[0];
    let bestError = Infinity;
    for (const thread of threads) {
      const error = totalError(pixels, [...chosen, thread]);
      if (error < bestError) { bestError = error; best = thread; }
    }
    chosen.push(best);
  }
  return chosen;
}

Each stitch then takes the closest of the chosen threads. A side effect I liked is that the choices are nested: the first 12 threads of a 30-colour run are a sensible 12-colour palette. That makes it natural to show versions with 12, 20 and 30 colours side by side and highlight the fewest that stay close to the image.

The bit nobody warns you about: buying the thread

A chart isn't finished until you know how much floss to buy. Stitches per skein depends on the cloth, because a coarser cloth uses more thread per stitch. With 2 strands, the tool assumes:

Cloth	Stitches per skein
11-count	1250
14-count	1500
16-count	1600
18-count	1700
28-count, over 2	1500

Isolated stitches waste more thread, because you travel to them and tie off. A stitch with fewer than three same-colour neighbours (out of eight) counts as scattered and costs 1.5 times as much. To keep the maths exact, the code works in half-stitch units:

const units = 2 * solid + 3 * scattered;
return Math.max(1, Math.ceil((units * strands) / (4 * STITCHES_PER_SKEIN[fabric])));

So 3,000 solid stitches on 14-count with 2 strands gives 6000 × 2 / (4 × 1500) = 2 skeins. These are estimates, since real usage varies with tension and how you travel, which is why the result always rounds up.

What I'd tell someone starting this

Convert to Lab first, and test CIEDE2000 against the published pairs.
Treat the thread list as the palette, rather than snapping to it at the end.
Let people compare colour counts. "How many colours?" is a budget question as much as a quality one.

If you want to see the steps from the stitcher's side, from cropping to the printed chart, here's the step-by-step guide. Everything runs in the browser, so the photo never leaves the device.

Checking a .pkpass in the browser: manifest hashes, the PKCS #7 signature, and what a green tick can't tell you

Silkline — Sat, 31 Oct 2026 14:00:00 +0000

In an earlier post I went through what's inside an Apple Wallet .pkpass file and how my browser viewer draws one. The section on manifest.json and signature ended with an admission: the viewer doesn't read either file.

This post is the follow-up. What does checking those two files take in a browser, what does a passing result actually prove, and why does the viewer still not do it?

All the code below is TypeScript that compiles under strict and that I ran against seven test files: passes I signed with a throwaway certificate authority using openssl smime, tampered copies of them, the unsigned sample pass from my own site, and one real pass from 2014, signed by Apple's chain, that sits in an open-source Android wallet's test files.

What the two files promise

manifest.json maps each file in the pass to the SHA-1 hash of its bytes, as lowercase hex. The 2014 pass, for example (hashes replaced):

{"icon.png":"<40 hex characters>","pass.json":"<40 hex characters>", ...}

signature is a DER-encoded PKCS #7 (CMS) SignedData structure. It is detached: the manifest's bytes are not inside it, you supply them. Together they say: these exact files, signed by the holder of this certificate.

So there are three checks, from cheap to expensive: do the files match the manifest, does the signature match the manifest, and who does the certificate belong to.

Step 1: unzip without skipping anything

My viewer unzips with fflate and quietly skips some entries, which is right for drawing a pass:

function isSkippedEntry(name: string): boolean {
  return (
    name.endsWith('/') ||
    name.startsWith('__MACOSX/') ||
    name.split('/').pop() === '.DS_Store' ||
    PROTOTYPE_KEYS.has(name)
  );
}

For checking, it's wrong. A .DS_Store or a stray file that isn't in the manifest is exactly what you want to report. But one rule has to stay: fflate returns the files as a plain object keyed by entry name, so an entry called __proto__ would land on the prototype. The way out is to record every name separately and only keep the dangerous ones out of the object:

export function readForCheck(bytes: Uint8Array): { files: Unzipped; names: string[] } {
  const names: string[] = [];
  const budget = new UnpackBudget();
  const files = unzipSync(bytes, {
    filter: (entry) => {
      if (entry.name.endsWith('/')) return false;
      names.push(entry.name);
      budget.spend(entry.originalSize);
      return !PROTOTYPE_KEYS.has(entry.name);
    },
  });
  return { files, names };
}

UnpackBudget is the viewer's existing size limit (60 MB uncompressed for one opened file), counted before each entry is inflated. A test pass with a __proto__ entry comes out as not in manifest: __proto__, and the object's prototype is untouched.

Step 2: the manifest, in both directions

crypto.subtle.digest supports SHA-1, so this needs no library. Note that crypto.subtle only exists on HTTPS pages (and localhost).

/** A copy as a plain ArrayBuffer, which is what WebCrypto's and PKI.js's types ask for. */
const ownBuffer = (bytes: Uint8Array): ArrayBuffer => new Uint8Array(bytes).buffer;

const toHex = (buffer: ArrayBuffer): string =>
  Array.from(new Uint8Array(buffer), (byte) => byte.toString(16).padStart(2, '0')).join('');

export async function checkManifest(files: Unzipped, names: string[]): Promise<string[]> {
  const manifestBytes = getFile(files, 'manifest.json');
  if (!manifestBytes) return ['no manifest.json'];
  const manifest: Record<string, string> = JSON.parse(new TextDecoder().decode(manifestBytes));
  const problems: string[] = [];
  for (const [path, expected] of Object.entries(manifest)) {
    const bytes = getFile(files, path);
    if (!bytes) {
      problems.push(`listed but missing: ${path}`);
      continue;
    }
    const actual = toHex(await crypto.subtle.digest('SHA-1', ownBuffer(bytes)));
    if (actual !== String(expected).toLowerCase()) problems.push(`hash mismatch: ${path}`);
  }
  for (const name of names) {
    if (name === 'manifest.json' || name === 'signature') continue;
    if (!Object.hasOwn(manifest, name)) problems.push(`not in manifest: ${name}`);
  }
  return problems;
}

getFile is the viewer's own-property lookup (Object.hasOwn), so a manifest that lists constructor can't resolve to something on Object.prototype.

Results on the test files: a pass where I changed one value in pass.json gives hash mismatch: pass.json; one with an extra file gives not in manifest: extra.txt. The real 2014 pass gives four listed but missing lines: the test copy kept only pass.json, but the manifest still lists the icon and logo images.

This step alone proves little. Anyone who edits pass.json can recompute its hash and write a new manifest. That's what the signature is for.

Step 3: the signature

Browsers have no CMS parser, so this uses PKI.js and its ASN.1 library, which do the crypto through WebCrypto:

export async function checkSignature(files: Unzipped) {
  const der = getFile(files, 'signature');
  const manifest = getFile(files, 'manifest.json');
  if (!der || !manifest) return { verified: false, reason: 'no signature or manifest' };
  try {
    const signedData = new pkijs.SignedData({ schema: pkijs.ContentInfo.fromBER(ownBuffer(der)).content });
    const result = await signedData.verify({ signer: 0, data: ownBuffer(manifest), extendedMode: true });
    const cert = result.signerCertificate!;
    const attributes = signedData.signerInfos[0].signedAttrs?.attributes ?? [];
    return {
      verified: result.signatureVerified === true,
      passTypeId: nameValue(cert.subject, OID.userId),
      teamId: nameValue(cert.subject, OID.orgUnit),
      signer: cert,
      validFrom: cert.notBefore.value,
      validTo: cert.notAfter.value,
      signingTime: attributes.find((a) => a.type === OID.signingTime)?.values[0].toDate() as Date | undefined,
      certificates: (signedData.certificates ?? []) as pkijs.Certificate[],
    };
  } catch (error) {
    return { verified: false, reason: String(error) };
  }
}

Because the signature is detached, the manifest goes in as data. verify hashes it with the signer's digest algorithm, compares that with the messageDigest attribute inside the signature, then checks the RSA signature over the signed attributes with the certificate's public key. The 2014 pass uses SHA-1 throughout; OpenSSL 3 signed my test passes with SHA-256. Both verify.

The try is not optional. When I changed the manifest without re-signing, PKI.js 3.4 didn't return false; it threw Message digest doesn't match.

Step 4: whose certificate is it?

A valid signature only means "someone with this certificate signed this manifest". The certificate's subject says who. On the 2014 pass it looks like this (names replaced):

UID=pass.<reverse domain>.ticket, CN=Pass Type ID: pass.<reverse domain>.ticket, OU=<10-character team ID>, O=<company name>, C=<country>

UID is the pass type identifier and OU is the Apple team identifier. Apple's documentation says passTypeIdentifier and teamIdentifier in pass.json must match the signing certificate, so compare them:

const OID = {
  userId: '0.9.2342.19200300.100.1.1', // UID: the pass type identifier
  orgUnit: '2.5.4.11', // OU: the team identifier
  signingTime: '1.2.840.113549.1.9.5',
};

const nameValue = (name: pkijs.RelativeDistinguishedNames, oid: string): string | undefined =>
  name.typesAndValues.find((item) => item.type === oid)?.value.valueBlock.value;

But nothing so far ties the certificate to Apple. I can make a certificate with any UID I like, and my throwaway test CA did exactly that.

Step 5: the chain to Apple

The signature in the 2014 pass carries two certificates: the pass certificate and the Apple Worldwide Developer Relations intermediate that issued it. The root is not included; you bring it. Apple publishes its root and intermediate certificates at apple.com/certificateauthority. That page lists several WWDR intermediates today (G2 to G6, with different expiry dates), so build the chain from the certificates in the signature up to a root you trust, rather than pinning one intermediate.

export async function checkChain(certificates: pkijs.Certificate[], appleRoot: pkijs.Certificate, at: Date) {
  const engine = new pkijs.CertificateChainValidationEngine({
    trustedCerts: [appleRoot],
    certs: certificates,
    checkDate: at,
  });
  const { result, resultMessage } = await engine.verify();
  return { ok: result, message: resultMessage };
}

Results:

Test	Result
2014 pass, Apple Root CA, at its signing time (2014-06-24)	valid
2014 pass, Apple Root CA, today	`The certificate is either not yet valid or expired`
2014 pass, my test root instead	`No valid certificate paths found`
My test passes, Apple Root CA	`No valid certificate paths found`

Two details matter here. The pass certificate expired on 2014-09-16, so "valid today" fails for any pass whose certificate has expired, which is normal for a ticket you kept. And the signing time is a signed attribute that the signer's machine wrote: it tells you when the signer says they signed, not a trusted timestamp.

The certificates also list CRL and OCSP addresses for revocation. Checking those is a network request to Apple, and this code doesn't make one.

Step 6: is it a pass certificate?

A path to Apple's root only says the Worldwide Developer Relations CA issued the certificate. That CA issues many kinds: app signing, push notifications, Apple Pay and more. The chain check doesn't care which kind it is.

Apple's WWDR Certification Practice Statement (version 1.29, section 4.11.11, "Pass Certificates") lists the extensions in a pass certificate, including an Apple custom extension with OID 1.2.840.113635.100.6.1.16. The statement doesn't say what the extension holds. In the 2014 certificate it is a UTF8String containing the pass type identifier, the same value as the UID. So check that it's there and matches passTypeIdentifier:

/** Apple custom extension listed in every pass certificate profile of Apple's WWDR CPS (4.11.11). */
const PASS_CERT_EXTENSION = '1.2.840.113635.100.6.1.16';

/** The pass type ID stored in that extension, or undefined if the certificate doesn't have it. */
export function passCertificateId(cert: pkijs.Certificate): string | undefined {
  const extension = cert.extensions?.find((item) => item.extnID === PASS_CERT_EXTENSION);
  if (!extension) return undefined;
  const inner = asn1js.fromBER(extension.extnValue.getValue());
  return inner.result instanceof asn1js.Utf8String ? inner.result.getValue() : undefined;
}

cert here is the signer that checkSignature returns in step 3. On the 2014 pass this returns the pass type identifier from pass.json. On my test passes it returns undefined, because my test CA never added the extension.

Two limits. I've seen the extension's contents in only one real pass certificate, so if a checker finds the OID but a value it can't read, report that instead of failing the pass outright. And the extension is only meaningful together with step 5: anyone can put any extension in a certificate their own CA signs.

What a green result proves, and what it doesn't

If all six steps pass, you know the files haven't changed since they were signed by someone holding a certificate that chains to Apple's root, carries Apple's pass certificate extension, and names that pass type and team.

You don't know:

That it's from who you think. Any Apple developer can get a pass certificate for their own team. Copy an airline's pass.json, change the name, set passTypeIdentifier and teamIdentifier to your own, sign it, and every check above passes. Only comparing the team ID with the airline's real one catches that, and a generic viewer doesn't know which team ID to expect.
That it's still valid. A pass can be refunded or voided after it was issued. The voided key is only what this copy says; the current state lives with the issuer.
That it will scan at the gate, or that Wallet will add it. The scanner checks the barcode against the issuer's records, and Wallet has its own checks.

Why the viewer still doesn't do it

Size is one reason. Bundled with esbuild and minified, the code above comes to about 410 KB, or 89 KB gzipped, almost all of it PKI.js and asn1js. That's a lot for a page whose job is to show a boarding pass.

The bigger reason is what a check would look like on screen. A green "signature valid" reads as "this ticket is genuine", and the list above shows it means much less. So the viewer opens a pass, draws it, and says on its pages that it doesn't check the signature. It opens the unsigned sample pass on my own site, which has a manifest.json but no signature, and it opened my tampered test pass without comment. If you build a checker, show what each step found, not one verdict.

Summary

Record every ZIP entry name for checking, but keep __proto__ and friends out of the object fflate returns.
Check the manifest both ways: every listed file matches, every file is listed.
The signature is detached CMS: pass the manifest bytes as data, and catch errors, since a digest mismatch can throw.
Compare the certificate's UID and OU with passTypeIdentifier and teamIdentifier.
Chain to Apple's root yourself, and decide which date you check against. The signing time is the signer's claim.
Then check the signer certificate carries Apple's pass certificate extension (1.2.840.113635.100.6.1.16) for the same pass type ID.
A passing result still doesn't say who the issuer should be, whether the pass was voided, or whether it will scan.

For what each file in a pass does, without the code, there's a plain-English guide to the .pkpass format on the viewer's site.

32 data APIs for Chinese social media and e-commerce, with real examples

Silkline — Fri, 30 Oct 2026 14:01:06 +0000

Disclosure: I build and sell every tool in this list (Silkline on the Apify Store). They are paid, pay-per-result Actors. This post was drafted by an AI agent from my own run data; the numbers are from runs I made on October 28, 2026.

If you need data from Chinese platforms, the usual problems are the same everywhere: the sites want a Chinese phone number to log in, requests are signed, pages are in Chinese, and the official APIs are closed to foreign companies. I spent the last three weeks turning 32 platforms into simple APIs. You send a keyword, a link or an ID, and you get flat JSON back. No account, cookies or proxies on your side.

Every tool runs on Apify, so you can start it from the web console, the API, Python or JavaScript, schedule it, and export to JSON, CSV or Excel. You pay per result (some tools add a small fee per run); each store page lists the prices.

Below are the 32 tools in four groups, one line each, then two real examples.

Chinese social media and content

Xiaohongshu (RedNote) scraper: notes by keyword with likes, saves and comments, comments and replies, creator profiles. Use it to see how Chinese consumers review a product before you launch it.
Douyin scraper: video search, creator videos and profiles, comments, hashtag and sound feeds, the hot search board. Use it to find which Douyin creators already talk about your category.
Weibo scraper: the live hot search list, post search by keyword and date, comments, reposts, user profiles. Use it for brand monitoring and crisis alerts.
Bilibili scraper: video search, comments, danmaku (bullet comments), subtitles, creator stats. Use it to read what gamers and tech fans say about a launch.
WeChat Official Account scraper: article search, an account's articles, full text, read, like and share counts, WeChat Channels video search. Use it to track what industry accounts publish and how much it is read.
Zhihu scraper: answers, articles, all answers to a question, comments, the hot list. Use it for long-form opinions on a product or a market.
Kuaishou scraper: videos, live rooms and hashtags by keyword, user videos, comments, hot lists. Use it to cover the lower-tier-city audience Douyin misses.
Douban scraper: hot movie and TV lists with scores, short comments, reviews, subject details. Use it to measure how a film or series lands in China.
Toutiao scraper: Toutiao articles and videos with full text, read and like counts, comments. Use it to follow news coverage of a company.
Youku scraper: video search, video details, channel profiles. Use it to check how a show or channel performs on Youku.
China social media search: one keyword across Weibo, WeChat, Zhihu, Douyin, RedNote, Bilibili, Kuaishou and news sites in one run. Use it as a daily brand-mention sweep.

E-commerce

Taobao and Tmall scraper: products by keyword with price, discount price, sales, shop ratings and ship-from location; buyer Q&A; shop catalogues. Use it for price research before sourcing.
1688 scraper: wholesale products with MOQ tier prices, order counts, supplier scores and factory verification. Use it to shortlist suppliers.
JD.com scraper: a shop's products and buyer reviews with star score, text and photos. Use it to read complaints about a competitor's product.
Douyin Shop scraper: products with price, 30-day sales, creator commission and how many creators promote each one. Use it for affiliate product selection.
TikTok Shop scraper: product search, details with SKUs, reviews, shop catalogues and hot-selling lists in nine markets. Use it to spot products that sell before you stock them.
AliExpress scraper: products by keyword with price, sold count and rating. Use it to compare cross-border prices.
Amazon scraper: search, categories and Best Sellers on 24 marketplaces with ASIN, price, rating and bought-past-month. Use it to check demand for a product you found in China.

Global platforms

TikTok scraper: videos by keyword, hashtag, profile or sound, creator search. Use it to collect a hashtag's top videos.
TikTok LIVE scraper: live room search, live status, the daily live creator ranking, top gifters. Use it to find live sellers in a niche.
YouTube scraper: Shorts and videos by keyword or channel, with views, duration and upload time. Use it to track a channel's output.
Instagram scraper: posts, Reels, profiles, comments, followers, hashtag and location feeds. Use it for influencer vetting.
Reddit scraper: posts by keyword, subreddit feeds, comments, user history. Use it to find threads where people ask for your product.
Threads scraper: posts by keyword, profile posts, replies. Use it to follow a brand's conversation on Threads.
Telegram scraper: public channel messages with views and reactions, search, subscriber counts. Use it to archive and compare channels.
IMDb scraper: the Top 250 movies and TV charts with rating and vote count in 10 languages. Use it as a clean reference dataset.

Marketing and creator (KOL) data

Xingtu KOL scraper: Douyin creators from the Xingtu (巨量星图) marketplace with video quotes, expected CPM and audience breakdown. Use it to budget a Douyin influencer campaign.
Pugongying KOL scraper: RedNote creators from the Pugongying (蒲公英) marketplace with note rates and median reads. Use it to shortlist RedNote creators by price.
Huahuo KOL scraper: Bilibili creators from the Huahuo (花火) marketplace with quotes, median plays and CPM. Use it to price a Bilibili sponsorship.
Douyin analytics: account audience insight, video performance, keyword heat index and trends. Use it to size interest in a keyword over time.
WeChat Index scraper: the WeChat Index (微信指数) of a keyword for the last 30 days. Use it to compare brands' popularity week by week.
TikTok Creative Center scraper: top ads by keyword and industry, trending hashtags, songs and search words. Use it to study competitors' ads.

Example 1: which sunscreen should a Douyin creator promote?

I searched Douyin Shop for 防晒霜 (sunscreen) and got 20 products. Sorted by 30-day sales, the top two were:

	30-day sales	Price	Commission	Creators promoting
#1	4,810	¥29.9	40%	273
#2	1,775	¥39.9	5%	183

The best seller pays eight times the commission of the runner-up and is cheaper. For a creator that is the obvious pick, and the 273 creators already on it show that others found it too. For a brand selling sunscreen on Douyin, the table shows what you compete against: 40% commission at ¥29.9.

The input was three fields:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("clover_folktale/douyin-shop-scraper").call(
    run_input={"mode": "search", "keywords": ["防晒霜"], "maxResults": 20}
)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
top = sorted(items, key=lambda x: -(x.get("monthlySales") or 0))[:2]
for p in top:
    print(p["monthlySales"], p["price"], p["commissionRate"], p["cooperatingCreators"])

Example 2: where do camping chairs on Taobao come from?

A Taobao search for 露营椅 (camping chair), best sellers first, 20 products:

16 of the 20 are Tmall stores, not small Taobao sellers.
18 of the 20 ship from two provinces: Zhejiang (11) and Hebei (7).
Six of them show 10,000+ sold. Prices after discount run from ¥15.8 to ¥139, median ¥37.9.

If you source outdoor furniture, that is a short list of where to look for factories, and a price floor to beat. The same run gives shop ratings and sales text for each item, so the next step is to search the same product in the 1688 scraper and compare wholesale prices.

run = client.actor("clover_folktale/taobao-scraper").call(
    run_input={"mode": "search", "keywords": ["露营椅"], "sort": "sales", "maxResults": 20}
)

Try them

On the Apify free plan each tool returns a stored sample of real results, so you can see the fields before paying; running your own input needs a paid Apify plan. Code examples and output samples for all 32 are in the examples repo on GitHub. If a field or a platform you need is missing, open an issue on the tool's page and tell me.

Reading Outlook .msg files in the browser: the compound file, MAPI properties, and a body that only exists as RTF

Silkline — Fri, 30 Oct 2026 11:58:57 +0000

An .eml file is text: headers, a blank line, then MIME parts. Open one in a text editor and you can read most of it. An Outlook .msg file is different. In an editor you see binary noise, with fragments of text that have a NUL byte between every letter.

I work on a browser tool that turns .eml and .msg emails into PDFs. This post is about the .msg side: what the file actually is, which part I left to a library, and the small decisions the library leaves to you.

A file system inside one file

A .msg is a Compound File Binary (CFB) file, also called OLE2 structured storage: the same container old .doc and .xls files use. Here are the first 32 bytes of the test file I use, a 23 KB Outlook message from msgreader's test data:

00000000: d0cf 11e0 a1b1 1ae1 0000 0000 0000 0000
00000010: 0000 0000 0000 0000 3e00 0300 feff 0900

D0 CF 11 E0 A1 B1 1A E1 is the CFB signature. 03 00 at offset 0x1A is major version 3, and 09 00 at 0x1E is the sector shift: sectors are 2⁹ = 512 bytes. Sectors are chained through a FAT, much like a floppy disk. Streams smaller than 4096 bytes are kept in a separate "mini stream" with 64-byte sectors, so a short subject doesn't waste a whole sector. A directory of 128-byte entries describes a tree of storages (folders) and streams (files).

This is that tree for the test file, shortened:

Root Entry/
  __properties_version1.0          864 bytes
  __substg1.0_0037001F              30 bytes   subject
  __substg1.0_0C1A001F              18 bytes   sender name
  __substg1.0_0C1F001F              52 bytes   sender address
  __substg1.0_007D001F            1164 bytes   transport headers
  __substg1.0_1000001F              38 bytes   plain-text body
  __substg1.0_10090102            1456 bytes   compressed RTF body
  __nameid_version1.0/              ...
  __recip_version1.0_#00000000/
    __substg1.0_3001001F                       display name
    __substg1.0_3003001F                       address
    __properties_version1.0
  __attach_version1.0_#00000000/
    __substg1.0_37010102           726 bytes   the attached file
    __substg1.0_3707001F                       long file name
    __substg1.0_370E001F                       MIME type
    __properties_version1.0
  __attach_version1.0_#00000001/    ...
  __attach_version1.0_#00000002/    ...

Stream names are property tags

Outlook stores a message as a bag of MAPI properties, and the stream name tells you which one you're looking at: __substg1.0_PPPPTTTT, where PPPP is the property ID and TTTT its type. 001F is a UTF-16LE string (that's where the NUL bytes come from), 001E an 8-bit string in the message's code page, 0102 binary. So 0037001F is PidTagSubject as a Unicode string.

Fixed-size values such as integers, flags and timestamps don't get streams. They sit in __properties_version1.0 as 16-byte records, after a 32-byte header at the top level and an 8-byte header inside each recipient or attachment storage. The test file has (864 − 32) / 16 = 52 of them at the top. IDs from 0x8000 up are named properties, which you resolve through __nameid_version1.0. All of this is in Microsoft's MS-OXMSG specification.

The part I didn't write

Reading CFB and decoding property streams is done by @kenjiuno/msgreader. Our own code is an adapter of about 160 lines that turns its output into the same object the .eml parser returns, so the preview and the PDF code don't care which format a message came from.

Loading the library had one surprise. It's a CommonJS module, and a dynamic import() hands back different shapes in Node and in the browser bundle:

/**
 * msgreader is a CommonJS module with `exports.default = MsgReader` and `__esModule`. Node and vitest hand back the class as
 * `default`; the Vite browser bundle hands back the whole exports object, so `default` is `{ default: MsgReader }`.
 */
async function loadMsgReader(): Promise<MsgReaderClass> {
  const loaded = (await import('@kenjiuno/msgreader')) as unknown as { default: MsgReaderClass | { default: MsgReaderClass } };
  return typeof loaded.default === 'function' ? loaded.default : loaded.default.default;
}

The adapter only depends on two methods, getFileData() and getAttachment(index), so most tests feed it plain objects. One test parses the real file above.

What the PDF header block shows, and where each value comes from:

Shown as	msgreader field	MAPI property
Subject	`subject`	`PidTagSubject` 0x0037
From	`senderName`, `senderEmail`, `senderAddressType`	0x0C1A, 0x0C1F, 0x0C1E
To, Cc	`recipients[]` with `recipType`	`PidTagRecipientType` 0x0C15
Date	`messageDeliveryTime`, then `clientSubmitTime`, then `creationTime`	0x0E06, 0x0039, 0x3007
Full headers page (optional)	`headers`	`PidTagTransportMessageHeaders` 0x007D

Exchange addresses

Inside an organization on Exchange, the sender's address type is often EX and the "address" is an internal directory name like /O=EXCHANGELABS/OU=.../CN=RECIPIENTS/CN=.... Printed on a PDF, that tells a reader nothing. Outlook usually stores an SMTP address next to it, so when the type is EX, the adapter looks for one in this order:

PidTagSenderSmtpAddress (0x5D01), which msgreader exposes as senderSmtpAddress. This is the documented property for the sender's SMTP address.
0x5D0A, which msgreader calls creatorSMTPAddress. It isn't in Microsoft's property list; it's the SMTP address of whoever created the item, which for a received message is usually the sender, but not always.
The EX address itself, if neither is there.

const EXCHANGE_ADDRESS_TYPE = 'EX';

function senderAddress(data: MsgData): string {
  const address =
    data.senderAddressType === EXCHANGE_ADDRESS_TYPE
      ? data.senderSmtpAddress || data.creatorSMTPAddress || data.senderEmail
      : data.senderEmail;
  return formatAddress(data.senderName, address);
}

function recipientAddress(recipient: MsgRecipient): { name?: string; address?: string } {
  const address = recipient.addressType === EXCHANGE_ADDRESS_TYPE ? recipient.smtpAddress || recipient.email : recipient.email;
  return { name: recipient.name, address };
}

The fallbacks use ||, not ??: a string property can come back as "" once its trailing NUL is stripped, and an empty string should not stop the search.

Recipients are grouped by type into To and Cc. A message you sent and then saved can also list Bcc recipients; the header block leaves them out.

Which date is "the date"

A .msg can carry several timestamps, and they mean different things:

function parseDate(data: MsgData): Date | null {
  const raw = data.messageDeliveryTime ?? data.clientSubmitTime ?? data.creationTime;
  if (!raw) return null;
  const date = new Date(raw);
  return Number.isNaN(date.getTime()) ? null : date;
}

We put delivery time first. On a received copy, PidTagMessageDeliveryTime is when it reached the mailbox, and a PDF saved as a record usually answers "when did we get this?". That isn't the only reasonable choice: when Outlook prints an email, its "Sent" line is the submit time (PidTagClientSubmitTime). The two differ when mail sits in a queue; if your PDFs should match Outlook's printouts, put clientSubmitTime first.

In the test file, delivery and submit time are both 00:48:31 UTC, but the creation time is 00:52:39: that's when this copy was saved, four minutes after the mail arrived. So creation time is only the last resort, for drafts that were never sent. The same goes for headers: a draft has no PidTagTransportMessageHeaders, so the optional full-headers page has nothing to show for it.

Three copies of the body

A message can hold its body up to three times: plain text in PidTagBody (0x1000), HTML in PidTagHtml (0x1013), and RTF in PidTagRtfCompressed (0x1009). The HTML often comes as bytes in the code page given by PidTagInternetCodepage (0x3FDE), which TextDecoder doesn't accept as a number. Most Windows code pages work as windows-<n>, but not the East Asian ones: windows-932, windows-936 and windows-950 aren't encoding labels, and new TextDecoder('windows-932') throws. So they get their own names:

const LABELS: Record<number, string> = {
  932: 'shift_jis',
  936: 'gbk',
  949: 'euc-kr',
  950: 'big5',
  65001: 'utf-8',
};

export function codepageLabel(codepage: number): string | undefined {
  const label = LABELS[codepage] ?? `windows-${codepage}`;
  return isKnownLabel(label) ? label : undefined;
}

/** Code page from the message, else the HTML's own <meta charset>, else UTF-8. */
function htmlCharset(bytes: Uint8Array, codepage: number | undefined): string {
  const fromCodepage = codepage ? codepageLabel(codepage) : undefined;
  if (fromCodepage) return fromCodepage;
  const head = new TextDecoder('latin1').decode(bytes.subarray(0, 2048));
  const declared = META_CHARSET.exec(head)?.[1];
  return declared && isKnownLabel(declared) ? declared : 'utf-8';
}

The test file declares code page 20127 (US-ASCII), which has no decoder under either name. That's fine: the fallbacks are the HTML's own <meta charset> and then UTF-8, and for ASCII text any of them gives the same result.

A body that only exists as RTF

The test file shows the catch. It has no 0x1013 stream at all. Its only formatted copy is the 1,456-byte compressed RTF. Decompressed, that's 3,073 bytes starting with:

{\rtf1\ansi\ansicpg932\fromhtml1 \fbidis \deff0{\fonttbl ...

\fromhtml1 means this RTF is a wrapper around HTML, as described in [MS-OXRTFEX]: the original tags sit in {\*\htmltag ...} groups, and RTF-only text sits between \htmlrtf and \htmlrtf0. Unwrap it and you get back the HTML Outlook's editor produced.

So when there's no HTML stream, the tool decompresses PidTagRtfCompressed and, if it wraps HTML, uses that. Plain RTF (no \fromhtml1) only fills in the text when PidTagBody is missing. The same decoder also reads winmail.dat attachments, where Outlook's "Rich Text" mail keeps its body in the same compressed RTF. The format ([MS-OXRTFCP], "LZFu") is a 16-byte header followed by an LZ77 stream with a 4 KB dictionary that starts out pre-filled with a fixed RTF prefix:

export function decompressRtf(data: Uint8Array): Uint8Array {
  if (data.length < HEADER_BYTES) throw new RtfError('compressed RTF too short');
  const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
  const rawSize = view.getUint32(4, true);
  const type = view.getUint32(8, true);
  const body = data.subarray(HEADER_BYTES);
  if (type === UNCOMPRESSED) return body.slice(0, rawSize);
  if (type !== COMPRESSED) throw new RtfError('unknown RTF compression');
  return lzfu(body, rawSize);
}

Each control byte covers the next eight items: a 0 bit is a literal byte, a 1 bit is a two-byte reference with a 12-bit dictionary offset and a 4-bit length (plus 2). A reference whose offset equals the current write position marks the end. Running this over the test file's RTF stream returns the 3,073 bytes above, and the preview shows the body in its original font and italics instead of a line of plain text.

One trap when turning that RTF back into text: non-ASCII bytes come in two forms. Most are escaped as \'hh, but in a double-byte code page the second byte of a character can be written as itself when it falls in the ASCII range. The Shift-JIS "テ" (0x83 0x65) appears as \'83e, and "本" (0x96 0x7B) as \'96\{. Decode each \'hh run on its own and those characters break in half. The tool keeps every byte of visible text, escaped or not, in one buffer and decodes it at the next control word or group edge. And don't read the RTF with TextDecoder('latin1') to get one character per byte: that label means windows-1252 in browsers, which maps 0x80 to U+20AC.

Attachments

Each __attach_version1.0_#NNNNNNNN storage is one attachment. The adapter picks the best name available and keeps the Content-ID, so cid: references in an HTML body can find their inline images:

function attachmentsOf(source: MsgSource, data: MsgData): MailAttachment[] {
  return (data.attachments ?? []).map((info, index) => {
    const { fileName, content } = source.getAttachment(index);
    return {
      fileName: fileName?.trim() || info.fileName || info.fileNameShort || `${NO_NAME}-${index + 1}`,
      mimeType: info.attachMimeTag || 'application/octet-stream',
      size: content.byteLength,
      contentId: info.pidContentId?.replace(/^<|>$/g, '') || null,
      data: content,
    };
  });
}

fileName is the long name (0x3707) and fileNameShort the old 8.3 name (0x3704). If both are empty, the attachment is called attachment-1, attachment-2 and so on, rather than disappearing.

An attachment can also be a whole email (an attached message). msgreader rebuilds it as a standalone .msg named after its display name, so it shows up in the attachment list as a .msg file. And if a message carries a winmail.dat, it's unpacked after parsing, for .msg and .eml alike: the files inside replace it in the attachment list.

It all stays in the tab

The file is read with File.arrayBuffer() and parsed in the page. Files over 50 MB are rejected, because bigger ones are slow or crash a phone browser. S/MIME-encrypted messages can't be opened, since the private key isn't there to decrypt them.

Summary

A .msg is a CFB container. Stream names are MAPI property tags: ID plus type.
Fixed-size properties live in __properties_version1.0, after a 32-byte header at the top level and 8 bytes in each recipient and attachment storage.
Prefer the SMTP address when the address type is EX.
Pick the date on purpose: we use delivery time, then submit time, because the PDF records when mail arrived; Outlook prints submit time as "Sent". Creation time is when the copy was saved.
Map PidTagInternetCodepage to a TextDecoder label yourself, with fallbacks; 932, 936 and 950 need their own names.
Check for a body that only exists as compressed RTF. Its HTML can be unwrapped from \fromhtml1, and double-byte text needs all its bytes decoded together.

If you have a .msg file and no Outlook, you can read it and save it as a PDF in the browser at emltopdf.net/msg-to-pdf. Each email's PDF is free.

How to get Taobao product data in Python (prices, sales, shops)

Silkline — Wed, 28 Oct 2026 10:04:25 +0000

Disclosure: I built the Taobao Scraper used in this tutorial. It's a paid Actor on Apify (pay per result), and the pricing is listed at the end.

Taobao (淘宝) and its premium side, Tmall (天猫), are where most of China shops online. If you source products, track competitor prices or study what Chinese consumers buy, the data you want is there: prices, discounts, how many units sold, which shops sell them and where they ship from.

Getting it yourself is painful. The site is in Chinese, search needs a logged-in account, and requests are signed. This tutorial skips that and gets Taobao data into a pandas DataFrame in a few lines.

What you'll build

A short script that:

searches Taobao for a keyword, best sellers first,
analyses prices, Tmall share and shipping locations with pandas,
narrows the search to Tmall stores in a price range,
reads the buyer Q&A (问大家) of the top product,
lists everything that product's shop sells.

Setup

You need Python 3.10+ and an Apify account. Copy your API token from Console → Settings → API & Integrations.

pip install apify-client pandas
export APIFY_TOKEN="your-token-here"

Step 1: search by keyword

The Actor has three modes: search, questions and shop. Start with search.

Search in Chinese. "phone case" works, but "手机壳" is what Chinese shoppers type and gives the results they see.

import os
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
actor = client.actor("clover_folktale/taobao-scraper")

run = actor.call(
    run_input={
        "mode": "search",
        "keywords": ["手机壳"],   # "phone case"
        "sort": "sales",          # best sellers first; also "general", "priceHighToLow"
        "maxResults": 40,
    },
    max_total_charge_usd=Decimal("0.50"),  # hard spending cap for this run
)

items = list(client.dataset(run.default_dataset_id).iterate_items())
print(len(items), "products")

maxResults is the total for the whole run, and paging is handled for you. max_total_charge_usd is a safety net: the run stops once that amount is spent.

This uses apify-client 3.x, where call() returns a Run object. On 1.x/2.x, use run["defaultDatasetId"] instead of run.default_dataset_id.

Each item is a flat dict:

{
  "type": "product",
  "itemId": "1234567890123",
  "url": "https://item.taobao.com/item.htm?id=1234567890123",
  "title": "Example phone case title",
  "price": 136.0,
  "discountPrice": 99.0,
  "discountType": "Example promotion",
  "salesText": "1万+",
  "salesCount": 10000,
  "commentCount": null,
  "isTmall": true,
  "shopId": "1659916565",
  "shopName": "Example flagship store",
  "sellerLevel": 18,
  "sellerPositiveRate": 99.6,
  "descriptionRating": 4.8,
  "serviceRating": 4.8,
  "shippingRating": 4.8,
  "itemLocation": "广东 东莞",
  "imageUrl": "https://example.com/main.jpg"
}

Values above are illustrative; missing fields are null (search results often have no commentCount). Prices are in yuan. salesText is what Taobao shows ("1万+" means 10,000+), and salesCount is that number as an integer, so "1万+" becomes 10,000, a floor, not an exact count.

Step 2: look at the market with pandas

import pandas as pd

df = pd.DataFrame(items)

top = df.sort_values("salesCount", ascending=False)[
    ["title", "price", "salesText", "shopName", "isTmall", "itemLocation"]
].head(5)
print(top.to_string(index=False))

print(df["price"].describe()[["min", "50%", "max"]])
print("Tmall share:", round(df["isTmall"].mean(), 2))
print(df["itemLocation"].value_counts().head(5))

When I ran it on 22 October 2026 for 手机壳, sorted by sales:

the top 5 products had all sold 10,000+ units;
prices ran from ¥9.80 to ¥605.88, with a median of about ¥70;
38 of the 40 best sellers (95%) were Tmall stores;
23 shipped from Shenzhen and 11 from Dongguan.

That last point is the kind of thing that is hard to see on the website and takes one line in pandas. Save the whole table with df.to_csv("taobao_phone_cases.csv", index=False).

Step 3: Tmall stores in a price range

run = actor.call(
    run_input={
        "mode": "search",
        "keywords": ["手机壳"],
        "tmallOnly": True,
        "minPrice": 30,
        "maxPrice": 100,
        "maxResults": 20,
    },
    max_total_charge_usd=Decimal("0.50"),
)
tmall = list(client.dataset(run.default_dataset_id).iterate_items())

One detail that matters: the price range applies to the price after discount (discountPrice), which is what shoppers actually pay. In my run all 20 products had a discounted price between ¥30 and ¥100, but five of them listed a higher price, one at ¥200 discounted to ¥100. Filter on discountPrice in your own analysis too.

Step 4: read the buyer Q&A

Taobao's 问大家 ("ask everyone") section is where buyers ask people who already bought the product. It's short, honest feedback. Pass item IDs or product links to mode questions:

top_item = df.sort_values("salesCount", ascending=False).iloc[0]["itemId"]

run = actor.call(
    run_input={"mode": "questions", "items": [top_item], "maxResults": 10},
    max_total_charge_usd=Decimal("0.50"),
)
questions = list(client.dataset(run.default_dataset_id).iterate_items())

for q in questions[:3]:
    print(q["question"], "|", q["answerCount"], "answers |", q["answers"][0]["text"])

For the best-selling case, the first three questions were (translated):

"Is the quality good? Will it scratch the phone?" 4 answers. First answer: "No."
"Does it protect the camera too?" 4 answers. "Yes, but the case is a bit stiff and hard to take off."
"Does it affect wireless charging in the car?" 5 answers. "No."

Each question record has question, askedAt, answerCount and answers (each with text and answeredAt).

Step 5: everything a shop sells

shop_id = df.sort_values("salesCount", ascending=False).iloc[0]["shopId"]

run = actor.call(
    run_input={"mode": "shop", "shops": [shop_id], "maxResults": 50},
    max_total_charge_usd=Decimal("0.50"),
)
catalog = list(client.dataset(run.default_dataset_id).iterate_items())
print(len(catalog), "products in this shop")

Shop records have the same fields as search results. The source doesn't return the shop's name in this mode, so shopName is empty there; keep it from the search step.

What it costs

The Actor is pay-per-event:

Event	Price
Run start	$0.03 per run, plus one more for each further keyword, item or shop the run reaches
Product or question returned	$8 per 1,000

The 40-product search in step 1 costs $0.03 + 40 × $0.008 = $0.35. Check the store page for the current price on your plan. On the free Apify plan, every run returns a sample dataset so you can see the output format; real searches need a paid plan.

Going further

Track prices over time. Save the step 1 input as a task and schedule it daily in the Apify Console. Comparing discountPrice day by day shows when sellers start promotions.
No-code exports. Every dataset downloads as JSON, CSV or Excel and connects to Google Sheets, Zapier and Make.
AI agents. The Actor is available through the Apify MCP server, so an agent can call it directly.
Other Chinese marketplaces. The same approach works for 1688 (wholesale prices and suppliers) and Douyin Shop (sales and creator commission rates).

The data is public listing information only. You are responsible for using it in line with applicable laws and Taobao's terms.

If a field you need is missing, open an issue on the Actor page. I read them.

Printing Data Matrix codes at an exact size: modules, DPI and PNG pHYs

Silkline — Sun, 25 Oct 2026 14:00:00 +0000

"Make the code 12 mm wide" sounds simple. Then the label goes through a 203 dpi thermal printer, the edges come out ragged, and the scanner struggles. The problem is rarely the barcode library. It's the step from millimetres to printer dots.

I recently built a browser Data Matrix generator for labels, and getting the physical size right was most of the work. This post covers the maths and the file formats, then GS1 input and a read-back check.

Modules and dots

A Data Matrix is a grid of square modules, for example 24×24, plus a quiet zone around it. If you want the code (without the quiet zone) to be 12 mm wide, each module is 12 / 24 = 0.5 mm.

A printer can't print 0.5 mm, though. It prints dots. At 203 dpi a dot is 25.4 / 203 ≈ 0.125 mm, so 0.5 mm is 3.996 dots. Print that as-is and each module edge gets rounded to a dot somewhere along the way, so modules come out at uneven widths. The fix is to round to a whole number of dots per module first, and derive every size from that:

export function resolvePrintSize(request: PrintRequest, { columns, rows, quietZone }: Geometry, referenceColumns = columns): PrintSize {
  const wantedModuleMm = request.basis === 'module' ? request.valueMm : request.valueMm / referenceColumns;
  const wantedDots = (wantedModuleMm * request.dpi) / MM_PER_INCH;
  const dotsPerModule = Math.max(1, Math.round(wantedDots));
  const moduleMm = (dotsPerModule * MM_PER_INCH) / request.dpi;
  return {
    dotsPerModule,
    moduleMm,
    totalWidthMm: (columns + 2 * quietZone) * moduleMm,
    pixelWidth: (columns + 2 * quietZone) * dotsPerModule,
    // ...heights, code size without quiet zone, and a flag if the module was raised to one dot
  };
}

For the 12 mm example at 203 dpi that's 4 dots, a 0.5005 mm module and a 12.01 mm code. At 300 dpi the same request becomes 6 dots per module and a 12.19 mm code: slightly bigger than asked, but every module is identical. The tool prints the actual module and code size, so the difference isn't hidden.

referenceColumns is for batches: codes with different data can get different symbol sizes, and passing the widest keeps one module size across the batch.

Getting the module grid out of bwip-js

bwip-js (a JavaScript port of BWIPP) does the encoding. I wanted the raw module grid rather than a canvas, so one matrix could feed SVG, PNG and PDF. bwip-js accepts a custom drawing object, so I pass one whose methods do nothing except record the outlines it traces:

polygon: (points: Point[]) => outlines.push(points.map(([x, y]) => [x, y] as Point)),
end: () => ({ outlines, width, height }),

At scale 1, one module is one unit, so testing the centre of each cell, (column + 0.5, row + 0.5), against those outlines with an even-odd rule gives the dark and light modules.

For plain text there's one encoding decision. Latin-1 text goes in as single bytes. Anything else is encoded as UTF-8 behind ECI 000026, so readers know which code page the bytes are in.

SVG: a viewBox in modules, a size in millimetres

SVG makes physical size easy. The viewBox is in modules, and width and height are written in mm:

`<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ${width} ${height}" ${size} shape-rendering="crispEdges" role="img">`

In print mode size is something like width="14.014mm" height="14.014mm": the 12 mm example at 203 dpi plus a two-module quiet zone on each side. Dark modules are merged into horizontal runs per row in one <path>. The background rectangle is optional, because laser and cutting software read it as one more shape to burn.

PNG: one pixel per dot, plus pHYs

A PNG has no physical size unless you write one. The pHYs chunk stores pixels per unit on each axis, with unit 1 meaning metres. I write the PNG myself (fflate for the zlib stream, plus a small CRC and chunk writer), so the bytes are the same in every browser and no canvas is involved:

/** pHYs chunk: pixels per metre in both directions, so print software and image editors know the physical size. */
function pixelDensity(dpi: number): Uint8Array {
  const data = new Uint8Array(9);
  const perMetre = Math.round(dpi / 0.0254);
  const view = new DataView(data.buffer);
  view.setUint32(0, perMetre);
  view.setUint32(4, perMetre);
  data[8] = 1;
  return chunk('pHYs', data);
}

300 dpi becomes 11811 pixels per metre. The chunk goes after IHDR and before IDAT, as the PNG spec requires. In print mode each module is exactly dotsPerModule pixels, so one image pixel is one printer dot and nothing gets resampled on the way to the printer.

The PNG is one RGBA buffer in memory, so sizes over 10,000 px a side or 40 megapixels are refused in favour of SVG or PDF.

PDF: the page is the code

For PDF I use jsPDF with unit: 'mm' and format: [totalWidthMm, totalHeightMm], so the page is exactly the code plus its quiet zone. Modules are vector rectangles from the same row runs as the SVG. Print at 100%, not "fit to page", and it's the size you asked for.

GS1: brackets in, FNC1 out

GS1 Data Matrix carries element strings such as (01)09506000134352(17)281231(10)LOT42. Each Application Identifier (AI) has a format, and some, like the GTIN in (01), end in a mod-10 check digit (weights 3 and 1 alternating from the right). The tool recomputes it and says what the digit should be.

For the common AIs, values are checked against the regex GS1 publishes, so a bad date gets a specific message such as "13 isn't a month" instead of a terse linter error. Less common AIs are still checked by BWIPP's own GS1 linter inside bwip-js.

The brackets are not encoded. The encoder (gs1datamatrix in bwip-js) adds FNC1 at the start and the group separators itself. So the most important cleanup is removing what users paste from other tools: <GS>, {GS}, [FNC1], \x1d, a leading ]d2. Left in, they'd be encoded as data, and you'd get a wrong code that still scans.

Read it back before offering it

The last step is a self-check. After encoding, a separate decoder reads the drawn modules the way a scanner would: strip the finder and timing patterns, walk the data modules in the placement order from ISO/IEC 16022, decode the codewords. Then it compares the result with what was asked for:

export function readBack(matrix: ModuleMatrix, request: EncodeRequest): ScannedSymbol | null {
  try {
    const scanned = scanMatrix(matrix);
    const data = request.mode === 'gs1' ? scanned.text.replaceAll('\u001d', '') : scanned.text;
    return data === expectedData(request) ? scanned : null;
  } catch {
    return null;
  }
}

If it doesn't match, the code isn't offered at all. The decoder is written independently of the encoder. In the test suite, the same symbols in every size also go through zxing-wasm, and optionally libdmtx, as independent decoders.

Summary

Round to a whole number of printer dots per module, then derive every size from that.
Write SVG with a viewBox in modules and width and height in mm.
Give PNGs a pHYs chunk (pixels per metre, unit 1) and make one pixel one printer dot.
Make the PDF page exactly the code plus quiet zone, and print at 100%.
For GS1, validate AIs and check digits, and strip any separators the user typed.
Decode what you drew and compare it with the input before you hand the file over.

If you need a Data Matrix at a real size in mm, the generator is free for single codes at data-matrix-generator.com, and it runs entirely in your browser.

Seating wedding guests with constraints: a small optimiser in a Web Worker

Silkline — Thu, 22 Oct 2026 13:00:00 +0000

Seating wedding guests sounds like a spreadsheet job until you write down the rules. The Smiths sit together. Uncle Bob and Aunt Carol must not share a table. The grandparents stay at table 1. Every table should be about as full as the others. Each rule is easy alone; together they pull against each other.

I recently built a browser seating chart planner, and the part that took longest was the optimiser behind its "Seat everyone" button. Here is how it works.

One cost for everything

The input is guests with a household (a free-text group column), tables with a capacity, pair rules ("together" or "apart"), and pinned guests who stay where the user put them. The optimiser turns this into indices and typed arrays, and rules into weighted edges between two guests.

Instead of a separate feasibility check, everything becomes one number to minimise. Hard rules get a weight that outweighs all the soft terms together:

export const DEFAULT_WEIGHTS: Weights = {
  hard: 10_000,      // per broken "together" / "apart" rule
  together: 100,     // per "together" rule, when it is set as a soft wish
  groupSplit: 100,   // per table a household uses beyond the fewest that could hold it
  groupSpread: 10,   // per table-number gap between the pieces of a split household
  balance: 4,        // per squared guest away from a table's fair share
  // ...plus tag, relation and table-preference terms
};

The balance term deserves a note. Each table's fair share is capacity × seatable guests / total seats, and the cost is the squared distance from it. That spreads guests over every table instead of leaving one nearly empty.

Scoring a move by its delta

A local search tries tens of thousands of moves, so rescoring the whole plan each time would be slow. The fix is to make every cost term belong to exactly one table, group, tag or rule. A move then only touches a few terms: sum them, apply the move, sum them again.

/** Moves guests[i] to targets[i] and returns the change in total cost. Undo by moving them back. */
export function moveGuests(p: Problem, s: State, guests: readonly number[], targets: readonly number[]): number {
  const from = guests.map((guest) => s.tableOf[guest]);
  const before = localCost(p, s, guests, targets);
  guests.forEach((guest, index) => seat(p, s, guest, targets[index]));
  return localCost(p, s, guests, from) - before;
}

localCost uses an epoch counter so each table, group and rule is visited once per call, even when two moved guests share it.

The search, in four steps

Pins first. Pinned guests are seated and never moved.
Greedy start. A household plus everyone tied to it by "together" rules is a block. Blocks go largest first, each to the table where it adds the least cost. A block that can't go in whole without breaking a hard rule is placed guest by guest.
Repair if needed. If everyone has a seat but a hard rule is still broken, the hard part is treated as graph colouring: tables are colours with a capacity, "apart" pairs are edges. DSATUR builds a colouring, then a tabu search works on the remaining conflicts. Its result is used only if it breaks fewer hard rules.
Simulated annealing on the full cost.

Each annealing step proposes one of four moves: one guest to another table, a whole block piece, a guest to the table of a household member seated elsewhere, or a swap of everyone movable at two tables.

function propose(ctx: Context, fixed: Uint8Array): Move | null {
  const { p, s, random, movable } = ctx;
  const guest = movable[randomIndex(random, movable.length)];
  const roll = random();
  const table = randomIndex(random, p.tableCount);
  if (roll < 0.35) return shiftOrSwap(ctx, [guest], table, fixed, false);
  if (roll < 0.67) return shiftOrSwap(ctx, blockPiece(ctx, guest, fixed), table, fixed, true);
  if (roll < 0.95) return proposeJoin(ctx, guest, fixed);
  return proposeExchange(ctx, s.tableOf[guest], table, fixed);
}

If the target table is full, a shift becomes a swap. Acceptance is the textbook rule: keep a better move, keep a worse one with probability exp(-delta / temperature). The temperature cools geometrically from 30 to 0.3, so early steps can break a household apart to make room and late steps only polish. The best seating seen is restored at the end.

Random numbers come from a seeded generator (mulberry32), so the same input and seed give the same seating unless the time budget cuts the run short. The run stops at a 250 ms budget by default, or early once the cost reaches the balance floor, the lowest balance cost any whole-number seating can have.

Exact solving for small plans

For up to 60 guests there's a second stage. The same cost is written as a mixed-integer linear program and solved by HiGHS compiled to WebAssembly, in the browser. The annealing result is the starting solution, so the solver only has to improve it or prove it optimal within 1.5 seconds.

Unless the solver fails, it ends with a proof that nothing costs less, or a proven lower bound. Larger plans get a cheap combinatorial bound instead, where a big gap means "not proven", not "far from optimal".

Keeping it off the main thread

A solve of over a second would freeze the page, so the optimiser runs in a module Web Worker. Cancelling is the simplest thing possible: terminate the worker, and start a fresh one next time.

return {
  outcome,
  cancel: () => {
    discardWorker(); // worker.terminate(); worker = null
    finish({ status: 'cancelled' });
  },
};

Each request carries an id and replies with another id are ignored, so a late answer can't overwrite a newer run.

The planner runs it twice. A quick run with the exact solver off is shown immediately. Then, for small plans, a full run goes in the background, and its result replaces the quick one only if it is strictly cheaper and the user hasn't changed anything meanwhile.

When the rules can't all be met

"Something went wrong" is the worst answer here, so the optimiser looks for causes that hold for every possible seating:

an "apart" pair that a chain of "together" rules ties to one table. A breadth-first search finds the chain so the message can name it.
a "together" group larger than the largest table.
more guests who must all sit at different tables than there are tables.
an odd ring of "apart" rules with only two tables: three guests who each must avoid the other two can't fit on two tables.
two pinned guests who contradict a rule.

Anything still broken that none of these explain is reported as unresolved, with both names. If there simply aren't enough seats, the planner adds tables first and says how many.

Summary

Put hard rules and soft goals in one cost, with the hard weight above all soft terms combined.
Make every term belong to one table, group or rule, and score moves by their delta.
Greedy start, colouring repair for hard rules, then seeded simulated annealing.
For small plans, an exact MILP solver in WebAssembly can prove the result optimal.
Run it in a Web Worker, cancel by terminating, match replies by id.
When the rules can't be met, explain why, by name.

If you're planning a wedding or any event with tables, the planner is free at seating-chart-generator.com, and the guest list never leaves your browser.

Bates numbering PDFs in the browser with pdf-lib: shrink the page, then read the numbers back

Silkline — Thu, 22 Oct 2026 10:58:24 +0000

A Bates number is a label printed on every page of a set of documents: usually a short prefix and a running number, like ABC-000251, ABC-000252, and so on. No two pages in the set share one, so anyone can point at a single page and find it again.

I added Bates numbering to a browser tool that turns .eml and .msg emails into PDFs. Stamping a string at the bottom of a page sounds like ten lines of pdf-lib. It wasn't. Here is what the code does and why.

The format: prefix, padding, start number

The prefix is optional and limited to A–Z, a–z, 0–9, space and - _ . /, up to 20 characters, because the number is drawn in Helvetica, a standard PDF font. A character Helvetica can't draw is refused up front, with its code point in the message.

The number gets at least six digits. If the last number in the batch needs more, every page in that batch gets the wider width, so the numbers still sort as text:

export function batesStamp(options: BatesOptions, totalPages: number): BatesStamp {
  const digits = Math.max(BATES_DIGITS, String(options.start + totalPages - 1).length);
  return { ...options, digits };
}

export function batesNumber(stamp: BatesStamp, pageIndex: number): string {
  return `${stamp.prefix}${String(stamp.start + pageIndex).padStart(stamp.digits, '0')}`;
}

A batch starting at 999990 with 20 pages runs from ABC-0999990 to ABC-1000009: seven digits on all 20 pages.

The start number (1 to 9999999) accepts 1,353, full-width digits and leading zeros. Paste SMITH-001353 while the prefix is SMITH- and the prefix is taken off, but only after the input fails as a plain number, so a digit prefix can't eat part of the number.

Where the number goes on someone else's page

The number sits bottom right, 22 points up, black, 8pt, shrinking to as little as 4pt on receipts and labels. An optional grey "Page x of y" goes in the middle and moves up a line if the two would come within 6 points.

"Bottom right" means as displayed. Other people's PDFs carry /Rotate and CropBoxes that differ from the MediaBox, so the visible area is the CropBox clipped to the MediaBox, and every point is mapped from displayed coordinates into unrotated page space, with the angle the text needs:

export function displayedPoint(width: number, height: number, rotation: number, across: number, up: number): FooterPlacement {
  const turn = ((rotation % 360) + 360) % 360;
  if (turn === 90) return { x: width - up, y: across, rotate: 90 };
  if (turn === 180) return { x: width - across, y: height - up, rotate: 180 };
  if (turn === 270) return { x: up, y: height - across, rotate: 270 };
  return { x: across, y: up, rotate: 0 };
}

The tests run this over portrait, landscape, all three rotations, crop boxes inside and sticking out of the sheet, MediaBoxes off the origin, scans and a 226×152 label, checking that the number reads upright at the bottom right.

Don't cover anything: shrink the content

The obvious approach is a white box with the number on top. But an attached PDF is someone's evidence, and a box can hide a line of it. So the tool shrinks the page content evenly, just enough to leave an empty strip at the bottom, keeping its top edge and horizontal centre:

export function makeRoomAtBottom(page: PDFPage, band: number): void {
  const box = visibleBox(page);
  const turn = page.getRotation().angle;
  const [across, down] = sideways(turn) ? [box.height, box.width] : [box.width, box.height];
  if (band <= 0 || band >= down) return;
  const factor = (down - band) / down;
  const fixed = displayedPoint(box.width, box.height, turn, across / 2, down);
  const move: Move = { factor, shift: { x: (1 - factor) * (box.x + fixed.x), y: (1 - factor) * (box.y + fixed.y) } };
  moveContent(page, move, box);
  moveAnnotations(page, move);
}

moveContent wraps the existing streams in q … cm … Q and clips to the visible box, so what the CropBox hid (the black edge of a scan) stays hidden. moveAnnotations applies the same transform to link Rect and QuadPoints.

The nasty detail: real content streams don't always balance their graphics state. An extra q leaves the wrapper open, so the number lands inside the shrink; an extra Q closes it early. So the content is counted first, extra qs are opened for stray Qs to close, and stray qs are closed before the wrapper's own Q.

Pages the tool rendered itself already have a wide enough bottom margin, so only copied PDF pages are shrunk.

Read the numbers back before the download

Computing the right numbers isn't the same as writing them. After stamping, the tool loads the bytes it's about to hand you and reads every page's number. The stamp is drawn last, so the check takes the last <hex> Tj in the last content stream, trusted only if the streams balance their q/Q:

function lastShownText(page: PDFPage): string | null {
  const streams = contentStreamBytes(page);
  const last = streams?.at(-1);
  if (!streams || !last || !isBalanced(streams.slice(0, -1)) || !isBalanced([last])) return null;
  const shown = [...new TextDecoder('latin1').decode(last).matchAll(SHOWN_TEXT)].at(-1);
  return shown ? fromHex(shown[1]) : null;
}

Each page is compared with its expected number, counting across all output files. One wrong page fails the export with the page and its source file named, and nothing is downloaded. Tests cover a repeat, a gap and missing pages.

A file that can't be numbered (encrypted, unreadable, empty, an undecodable page) is listed with the reason and skipped. Files are also rechecked just before numbering, so one that changed after you added it can't shift everyone else's numbers.

Emails and their attachments in one sequence

The merged email export sorts oldest first; undated emails go last, in the order they were opened:

export function sortByDate(emails: readonly ParsedEmail[]): ParsedEmail[] {
  const time = (email: ParsedEmail) => email.date?.getTime() ?? Number.POSITIVE_INFINITY;
  return emails
    .map((email, index) => ({ email, index }))
    .sort((a, b) => time(a.email) - time(b.email) || a.index - b.index)
    .map(({ email }) => email);
}

Each email's PDF and image attachments become pages right after it (GIF, WebP and BMP via a canvas); other types are only named. A contents page goes in front without a number and lists where each email starts; the numbers begin on the first email and run through every page after it, so an email and its attachment run on without a break. A CSV lists begin and end numbers per email and attachment, with formula-looking cells escaped.

Remembering where you stopped

Productions come in batches. After a numbered export, nextStart = start + pages is saved to localStorage and offered as a "Start the next batch at N" button:

private offerNextBatch(range: BatesRange | null): void {
  if (!range || range.nextStart > BATES_START_MAX) return;
  this.nextStart = range.nextStart;
  this.startBoxAtExport = this.ui.batesStart.value;
  saveNumbering({ start: String(range.nextStart) });
  this.syncNextBatch();
}

Only the prefix, start number and a few checkbox choices are saved, never file content, and each is type-checked on read. Blocked or broken storage just means the page forgets on reload. The start box only changes if you haven't edited it since the export.

It all stays in the tab

Files are read with File.arrayBuffer(), pdf-lib loads when the first PDF is dropped, and the result leaves through a Blob URL. No server step.

Stamping your own PDFs is free for everyone. Numbering emails together with their attachments in one step, sorted by date, is part of the paid Case Pass and Legal plans.

Summary

Fix the digit width once per batch, so numbers sort as text.
Place text in displayed coordinates. Rotation and CropBox are normal in other people's PDFs.
Shrink the content to make room instead of covering it, and move the link annotations with it.
Count q/Q before you wrap someone else's content stream.
Read the numbers back from the final bytes before you let anyone download them.

If you need to number a set of PDFs without installing anything, the tool is at emltopdf.net/bates-numbering.

This post was written with AI assistance.

What's inside a .pkpass file, and how to render one in the browser

Silkline — Mon, 19 Oct 2026 13:00:00 +0000

A .pkpass file is an Apple Wallet pass: a boarding pass, a concert ticket, a gym card. On an iPhone it opens straight into Wallet. On Windows or Android it is an attachment nothing wants to open.

I recently built a small viewer that opens these files in the browser without uploading them. Most of the work was understanding the format, so here is what's inside a pass and how each part can be read.

It's a ZIP file

Rename a .pkpass to .zip and you can open it. Inside you'll find:

pass.json: the content of the pass, including the barcode data
images such as icon.png and logo.png, often with @2x and @3x versions
optional en.lproj/, de.lproj/ folders with a pass.strings file each
manifest.json and signature

The file comes from a stranger, so unzipping is the first place to be careful. I use fflate, check the ZIP magic bytes, and count uncompressed sizes in the filter callback, which runs before an entry is inflated:

export function readZip(bytes: Uint8Array, budget = new UnpackBudget()): Unzipped {
  if (bytes.byteLength > MAX_INPUT_BYTES) throw new PkpassError('too-large');
  if (!hasZipSignature(bytes)) throw new PkpassError('not-zip');
  return unzipSync(bytes, {
    filter: (entry) => {
      if (isSkippedEntry(entry.name)) return false; // directories, __MACOSX/, .DS_Store
      if (entry.originalSize > MAX_ENTRY_BYTES) throw new PkpassError('unpacked-too-large');
      budget.spend(entry.originalSize);
      return true;
    },
  });
}

One detail that's easy to miss: fflate returns files as a plain object keyed by entry name. An entry named __proto__ would collide with the prototype, so those names are skipped and every lookup uses Object.hasOwn.

pass.json: style and fields

A pass has one style, and the style is a top-level key, not a type field:

export const PASS_STYLES = ['boardingPass', 'eventTicket', 'coupon', 'storeCard', 'generic'] as const;

function detectStyle(json: JsonObject): PassStyle {
  const style = PASS_STYLES.find((name) => isObject(json[name]));
  if (!style) throw new PkpassError('not-a-pass');
  return style;
}

Inside it are five field lists: headerFields, primaryFields, secondaryFields, auxiliaryFields and backFields. Values need formatting. A date with dateStyle: "PKDateStyleShort" maps onto Intl.DateTimeFormat's dateStyle. ignoresTimeZone: true means "show the wall-clock time as written", so I strip the offset and format in UTC. Numbers with a currencyCode go to Intl.NumberFormat. attributedValue can hold HTML links; I reduce it to plain text, and every string from the pass reaches the page through textContent.

Localization: .lproj and .strings

Translations live in <language>.lproj/pass.strings, an old Apple format of "key" = "value"; pairs. Apple's docs say non-ASCII .strings files use UTF-16, so check for a byte-order mark before decoding:

export function decodeStringsFile(bytes: Uint8Array): string {
  if (startsWith(bytes, UTF16_LE_BOM)) return new TextDecoder('utf-16le').decode(bytes.subarray(2));
  if (startsWith(bytes, UTF16_BE_BOM)) return new TextDecoder('utf-16be').decode(bytes.subarray(2));
  return decodeUtf8(bytes);
}

You also need a rule for picking one language. Mine is en, then any en-*, then the first folder alphabetically. A missing key falls back to the untranslated text, and the parser keeps every pair it read before a syntax error.

Images and @2x / @3x

There are six image roles: logo, icon, strip, thumbnail, background and footer, each at up to three scales. Apple says an image should live either at the top level or inside a .lproj folder, never both. Some passes break that rule anyway, so the viewer picks one predictable order and checks the top level first:

const IMAGE_SCALES = ['@2x', '@3x', ''];

function findImage(files: Unzipped, name: PassImageName, language?: string): Uint8Array | undefined {
  const folders = language ? ['', `${language}.lproj/`] : [''];
  for (const folder of folders) {
    for (const scale of IMAGE_SCALES) {
      const bytes = getFile(files, `${folder}${name}${scale}.png`);
      if (bytes) return bytes;
    }
  }
  return undefined;
}

Before an image reaches the browser, I read its width and height from the PNG's IHDR chunk without decoding it. Non-PNGs, files over 5 MB and anything over 2000×2000 pixels are skipped, so a crafted image can't make the tab allocate a huge bitmap.

manifest.json and signature

manifest.json maps every file in the pass to its SHA-1 hash, and signature is a PKCS #7 detached signature of the manifest, made with the issuer's certificate. That's why you can't edit a pass: change one byte and the hashes no longer match what was signed.

My viewer doesn't read either file. It draws what pass.json says. That keeps it simple, but any tool like this should say the consequence plainly: a pass that renders fine is not proof that it's genuine or that Wallet will accept it.

Barcodes

The barcode lives in barcodes, an array of { format, message, messageEncoding, altText }. Wallet uses the first one it can display; older passes use a single barcode object. So I take the first supported entry of barcodes, then fall back to barcode. All eight formats map onto bwip-js:

export const BARCODE_SYMBOLOGIES = {
  PKBarcodeFormatQR: 'qrcode',
  PKBarcodeFormatPDF417: 'pdf417',
  PKBarcodeFormatAztec: 'azteccode',
  PKBarcodeFormatCode128: 'code128',
  PKBarcodeFormatCode39: 'code39',
  PKBarcodeFormatCodabar: 'rationalizedCodabar',
  PKBarcodeFormatEAN13: 'ean13',
  PKBarcodeFormatI2of5: 'interleaved2of5',
} as const;

The eight encoders are imported by name from bwip-js/browser, so the bundler can drop the rest, and loaded with import() on first use.

messageEncoding matters for non-ASCII text. bwip-js turns text into UTF-8 unless binarytext: true is set, in which case each character is one byte, which is ISO-8859-1. So: ISO-8859-1 messages that fit in Latin-1 go in as single bytes, everything else as plain UTF-8 with no ECI header, since older gate scanners may not understand one.

For the gate there's a full-screen view. It re-encodes the barcode at the largest whole number of device pixels per module that fits, so module edges stay sharp, and asks for a screen wake lock so the phone doesn't dim in the queue.

.pkpasses bundles

A .pkpasses file is a ZIP of .pkpass files, so opening one is the same function with one more branch:

export function openPassFile(bytes: Uint8Array): OpenResult {
  const budget = new UnpackBudget();
  const files = readZip(bytes, budget);
  if (getFile(files, PASS_JSON)) return { kind: 'pass', pass: parsePassArchive(files) };
  const items = readBundle(files, budget);
  if (items.length > 0) return { kind: 'bundle', items };
  throw new PkpassError('missing-pass-json');
}

The same budget covers the outer ZIP and every pass inside, so nesting can't get around the size limit. A broken pass becomes an error entry instead of failing the bundle.

Summary

A .pkpass is a ZIP. Check sizes before inflating and look files up with Object.hasOwn.
The style is whichever of five top-level keys exists. Field values need Intl formatting.
.strings files may be UTF-16 with a BOM. Fall back to the untranslated text.
An image belongs at the top level or in a .lproj folder, not both; pick a fixed lookup order for passes that break the rule. Read PNG dimensions before decoding.
manifest.json and signature protect the pass from edits. If you don't verify them, say so.
bwip-js covers all eight PassKit formats, and messageEncoding maps onto binarytext.

If you need to look at a pass on a computer or an Android phone, the viewer is free at pkpassviewer.com, and the file never leaves your browser.

11 scrapers for Chinese platforms in 8 days, and my first $4

Silkline — Wed, 14 Oct 2026 00:09:59 +0000

Disclosure: I build and sell the Actors mentioned here (Silkline on Apify). Most of the code was written with an AI coding agent; the product calls, pricing and checks were mine. Numbers are from my Apify console on October 12, 2026.

On October 5 I published three scrapers on the Apify Store: Bilibili, Weibo and Xiaohongshu (RedNote). Five days later there were eleven. On day 8 someone I've never met paid for the RedNote one.

Revenue so far: $4.10. Platform cost: $0.01.

That's not a business yet. But it answers the only question I cared about in week one: will anyone pay for this? Here's what got me there and what I'd tell anyone thinking about selling Actors.

Why Chinese platforms

I scanned the whole Apify Store (about 12,000 Actors) and looked at monthly users per platform. Chinese platforms stood out: real demand, few maintained tools, and a language barrier that keeps most builders away. RedNote, Douyin, Bilibili, Weibo and WeChat are where Chinese consumers review products, follow creators and argue about brands. Marketers and researchers outside China want that data and can't easily read the sites.

I don't run my own crawlers. The data comes from a commercial data API whose terms let me resell it, and I got that permission in writing before going public. The Actors turn it into clean, documented JSON with pricing per result.

The lineup today:

Xiaohongshu / RedNote: notes, comments, creator profiles
Douyin: videos, comments, hot search
Bilibili: videos, comments, danmaku, subtitles
Weibo: posts, comments, hot search
WeChat Official Accounts: articles, read counts, Channels video search
TikTok Shop, TikTok, YouTube, Reddit, Threads, Telegram

Lesson 1: "published" is not "found"

For the first few days the store search didn't show my Actors at all for their own platform names, and I assumed that was normal for new listings. It wasn't. They were listed but ranked far below the first page.

What actually moves ranking on Apify is the Actor quality score. The base items are mostly boring work: a README with real examples, an input schema with defaults that work, a dataset schema, an icon, and example tasks. I published 46 example tasks across the 11 Actors. Six Actors now sit at a quality score of 81, which seems to be about the ceiling before real users, bookmarks and reviews count.

Result after a week: the WeChat Actor went from roughly #46 to #30 for "wechat". Most of the others are still outside the top 100. Ranking is the main bottleneck, not code.

Lesson 2: compete on fields, not price

My first instinct was to undercut. I didn't. Pricing sits in the same range as the established Actors, and the difference is in the output.

The clearest example: for every commenter, author and reviewer, the Actors return the public profile fields the platform already shows: user ID, avatar, IP location (Chinese platforms display "IP属地" next to every comment), gender and verification. I made that the headline of every listing instead of a footnote. For anyone doing audience or regional analysis, it's often the column that matters most.

Lesson 3: price per result, charge only on success

All 11 Actors are pay-per-event:

a small run start fee, charged only after the first page of data actually arrives,
a per-result fee (a RedNote search result is $0.005),
and a hard cost cap per run, so a bad input can't burn money on either side.

If the upstream fails at the very start, the user pays nothing and the run is marked failed. It's more code than a flat price, but it means a user's first run can't go badly for them.

Lesson 4: your first paying user will find your bugs

The day the first payment showed up, my own daily self-check flagged the RedNote search mode: some results had an empty title.

The cause: many RedNote posts simply have no separate title, only body text. The API returned an empty title and I passed it through. Technically correct, practically useless in a spreadsheet. The fix was to fall back to the first line of the body (up to 50 characters), keep the full text in description, and ship a new build the same afternoon. A live run on 20 coffee posts came back with zero empty titles.

Lesson: run your own Actor on boring real queries every day, not just your test fixtures.

Lesson 5: memory is your margin

Apify charges the developer for compute on paid users' runs, out of your share. I launched everything at 4096 MB. Measured peak usage was about 100 MB. Dropping all 11 to 512 MB cut compute per run to roughly an eighth. On pay-per-result pricing, that's the difference between a margin and a loss.

The stack

Python, one shared client package vendored into each Actor at build time
Tests built on real recorded responses, so schema drift shows up as a failing test
A daily report script: users, runs, failures, store rank per keyword, upstream balance and now revenue
One person deciding, one AI agent writing most of the code and running the checks

What's next

Taobao/Tmall product data (waiting on a data provider)
A Product Hunt launch this week
Weekly progress posts here and on X (@silklinedata)

If you need data from Chinese platforms, the RedNote Actor that made the first sale is here: Xiaohongshu (RedNote) Scraper. If you're building Actors yourself, ask me anything about the quality score or pay-per-event pricing in the comments.

Previewing untrusted .eml files in the browser: sandboxed iframe, CSP and blocked remote images

Silkline — Mon, 12 Oct 2026 13:15:39 +0000

An .eml file is a saved email. Open one and you get raw MIME: headers, a plain-text part, an HTML part, maybe some base64 attachments. The HTML part is the interesting one, because it was written by whoever sent the email. It can contain scripts, forms, tracking pixels, CSS that pulls images from a server, and links that try to take over the page.

I recently built a small tool that previews .eml and Outlook .msg files in the browser and turns them into PDFs, without uploading anything. The preview was the part that needed the most care. If you just parse the email and drop its HTML into your page with innerHTML, you are running a stranger's markup inside your own origin.

Here is the setup I ended up with. It is three locks, each covering what the previous one might miss.

Lock 1: sanitize the HTML before it goes anywhere

The first step is DOMPurify. On top of its defaults I forbid a list of tags that have no business in an email preview:

const FORBIDDEN_TAGS = [
  'svg', 'math', 'form', 'input', 'button', 'select', 'textarea', 'option', 'iframe', 'frame', 'frameset',
  'object', 'embed', 'applet', 'link', 'meta', 'base', 'video', 'audio', 'source', 'track', 'picture', 'canvas', 'dialog',
];
const FORBIDDEN_ATTRIBUTES = ['srcset', 'formaction', 'ping', 'autofocus', 'poster'];

meta and base matter more than they look: a <meta http-equiv="refresh"> can redirect, and a <base href> changes where every relative link points.

Two hooks do the email-specific work:

Images. cid: references (images embedded in the email itself) are resolved to data URLs from the attachments. Any http(s):// image is replaced with a small grey placeholder, and the count is shown to the user with a "Load remote images" button. Remote images are how senders know you opened the mail, so they stay off until you ask.
Links. Every <a href> gets target="_blank" and rel="noopener noreferrer nofollow", so a click opens a new tab and the email can't reach back into the page that opened it.

Inline style attributes and <style> blocks get their url(https://...) values and @import rules stripped while remote images are blocked. That is the part where sanitizing alone starts to feel shaky: CSS is a whole language, and a regex over it is a best effort, not a guarantee.

Lock 2: render it in a sandboxed iframe

The cleaned HTML never touches the main document. It goes into an iframe through srcdoc:

const frame = document.createElement('iframe');
frame.setAttribute('sandbox', 'allow-same-origin allow-popups allow-popups-to-escape-sandbox');
frame.setAttribute('referrerpolicy', 'no-referrer');
frame.srcdoc = frameDocument(cleanHtml, loadRemoteImages);

What matters is what is not in that sandbox list: no allow-scripts, no allow-forms, no allow-top-navigation, no allow-modals. Even if something slipped past DOMPurify, it could not run a script, submit a form or navigate the tab. The popup flags are there only so a link the reader clicks can open in a normal new tab.

Lock 3: a CSP inside the iframe

The document written into srcdoc carries its own Content Security Policy:

function policy(loadRemoteImages: boolean): string {
  const images = loadRemoteImages ? 'img-src data: blob: https:' : 'img-src data: blob:';
  return `default-src 'none'; ${images}; style-src 'unsafe-inline'`;
}

default-src 'none' means the frame can't fetch anything except what is listed. With remote images blocked, only data: and blob: images load, so a background image hidden in CSS that the sanitizer missed still goes nowhere. This is the lock that makes the CSS regex in Lock 1 a convenience rather than the last line of defence. The parent page has its own CSP as well (script-src 'self'), so there are two policies between the email and the network.

The trade-off: a preview that grows with the email

The first version of the preview was a fixed-height box. Short emails sat at the top of a big empty frame, which looked broken. Making the iframe fit its content means the parent page has to read the height of the document inside it, and you can't read a sandboxed frame's document unless the sandbox includes allow-same-origin.

allow-same-origin sounds alarming, and combined with allow-scripts it is: a script inside the frame could then remove its own sandbox. Without allow-scripts, though, nothing inside the frame can execute, so being same-origin gives the email no new powers. It only lets my page measure it:

const fit = (): void => {
  frame.style.height = '0px'; // shrink first: scrollHeight never reports less than the frame's own height
  frame.style.height = `${Math.min(doc.documentElement.scrollHeight + 2, FRAME_MAX_HEIGHT)}px`;
};

Long emails stop at a maximum height and scroll inside the frame. The height is recalculated when images inside the email finish loading and when the card changes width.

The risk with this kind of setup is not today's code. It's someone, possibly me in six months, adding allow-scripts to fix an unrelated bug. So there is a test that reads the source and fails if the sandbox ever includes a dangerous flag:

it('never allows scripts, forms or top navigation', () => {
  for (const value of values) {
    for (const flag of ['allow-scripts', 'allow-forms', 'allow-top-navigation', 'allow-modals']) {
      expect(value.split(/\s+/)).not.toContain(flag);
    }
  }
});

It is a blunt test, and that is the point. The comment next to the sandbox line explains why allow-same-origin is there, and the test makes sure the reason stays true.

Parsing happens in the browser too

None of this would matter much if the file went to a server first. Parsing is done client-side: postal-mime for .eml, @kenjiuno/msgreader for Outlook .msg files, and pdfmake to build the PDF. You can check it the usual way: open the Network tab in DevTools, pick a file, and watch that no request carries it anywhere.

Summary

Sanitize with DOMPurify, forbid the tags an email doesn't need, rewrite images and links in hooks.
Render in a srcdoc iframe with a sandbox that has no allow-scripts.
Put a default-src 'none' CSP inside the frame, so anything the sanitizer misses still can't load.
If you add allow-same-origin to size the frame, write a test that keeps allow-scripts out.

I wrapped all of this into a free tool at emltopdf.net, if you ever need to open an .eml file without installing a mail client.

How to get Xiaohongshu (RedNote) data in Python

Silkline — Tue, 06 Oct 2026 06:51:13 +0000

Disclosure: I built the Xiaohongshu Scraper used in this tutorial. It's a paid Actor on Apify (pay per result), and the pricing is listed at the end.

Xiaohongshu (小红书), known abroad as RedNote, is where Chinese consumers go to decide what to buy. People post reviews, "what I bought" hauls, travel guides and café lists, and other people save those notes to come back to later. For brand research, product research or a Chinese-language NLP dataset, that's very valuable data.

It's also hard to get. The site is in Chinese, logged-out web access is restricted, requests are signed, and there's no public API. This tutorial skips all of that and gets Xiaohongshu data into Python in about 20 lines.

What you'll build

A short script that:

searches Xiaohongshu for a keyword,
loads the results into a pandas DataFrame,
ranks notes by saves (collects),
pulls the public comments of the top note.

Setup

You need Python 3.10+ and an Apify account. Copy your API token from Console → Settings → API & Integrations.

pip install apify-client pandas
export APIFY_TOKEN="your-token-here"

Step 1: search notes by keyword

The Actor has four modes: search, note, comments and user. We start with search.

One tip before you write any code: search in Chinese. "coffee" and "咖啡" return very different results, and the Chinese keyword is what real users type.

import os
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
actor = client.actor("clover_folktale/xiaohongshu-scraper")

run = actor.call(
    run_input={
        "mode": "search",
        "keyword": "咖啡",      # "coffee"
        "maxResults": 40,
    },
    max_total_charge_usd=Decimal("0.50"),  # hard spending cap for this run
)

items = list(client.dataset(run.default_dataset_id).iterate_items())
print(len(items), "notes")

maxResults is the total number of items for the whole run, and pagination is handled for you. max_total_charge_usd is a safety net: the run stops once that amount is spent.

This uses apify-client 3.x, where call() returns a Run object. On 1.x/2.x, use run["defaultDatasetId"] instead of run.default_dataset_id.

Each item is a flat dict. A search result looks like this (values illustrative):

{
  "type": "note",
  "noteId": "6aa72fd3000000002802c3f4",
  "url": "https://www.xiaohongshu.com/explore/6aa72fd3000000002802c3f4",
  "noteType": "image",
  "title": "Example title",
  "description": "Example description",
  "authorName": "ExampleCreator",
  "authorId": "63ebbadf00000000260055d5",
  "authorUrl": "https://www.xiaohongshu.com/user/profile/63ebbadf00000000260055d5",
  "publishedAt": "2026-09-14T10:00:51Z",
  "likes": 1658,
  "collects": 527,
  "comments": 131,
  "shares": 306,
  "coverUrl": "https://example.com/cover.webp",
  "imageCount": 9,
  "durationSeconds": null,
  "hashtags": []
}

Missing fields are null. durationSeconds is filled for video notes in search results. hashtags is filled only in note details (mode note), so it's an empty list here.

Step 2: rank by saves

Likes on Xiaohongshu are cheap. Collects (saves) mean someone plans to come back to the note, which usually makes it a guide, list or review. The save-to-like ratio is a quick way to find "useful" content.

import pandas as pd

df = pd.DataFrame(items)
df["save_ratio"] = df["collects"] / df["likes"].where(df["likes"] > 0)

top = df.sort_values("collects", ascending=False)[
    ["title", "authorName", "likes", "collects", "comments", "save_ratio", "url"]
].head(10)
print(top.to_string(index=False))

From here you can write df.to_csv("xhs_coffee.csv", index=False) and hand it to a marketing team, or keep going.

Step 3: read the comments of the top note

Comments are where the product feedback is. Pass note URLs or 24-character note IDs to mode comments:

top_note = top.iloc[0]["url"]

run = actor.call(
    run_input={"mode": "comments", "notes": [top_note], "maxResults": 100},
    max_total_charge_usd=Decimal("0.50"),
)
comments = list(client.dataset(run.default_dataset_id).iterate_items())

for c in comments[:5]:
    print(c["likes"], c["text"])

A comment item:

{
  "type": "comment",
  "commentId": "6a1f0c2e000000001e03a9b7",
  "noteId": "6aa72fd3000000002802c3f4",
  "text": "Example comment text",
  "likes": 42,
  "replyCount": 3,
  "publishedAt": "2026-09-15T08:12:40Z",
  "authorName": "ExampleUser"
}

Comment items include only text, likes, reply count, time and the commenter's public nickname. Commenter IDs, avatars and other personal data are never returned, which keeps your dataset simpler to store and share.

Step 4 (optional): creator profiles

To vet an influencer, use mode user with profile links or user IDs. You get one profile record followed by that creator's recent notes:

run = actor.call(
    run_input={
        "mode": "user",
        "users": ["https://www.xiaohongshu.com/user/profile/63ebbadf00000000260055d5"],
        "maxResults": 30,
    },
    max_total_charge_usd=Decimal("0.50"),
)
records = list(client.dataset(run.default_dataset_id).iterate_items())
profile = next(r for r in records if r["type"] == "user")
notes = [r for r in records if r["type"] == "note"]
print(profile["nickname"], profile["followers"], profile["likesReceived"], len(notes))

The profile record has userId, url, nickname, bio, followers, following, noteCount, likesReceived and verified.

To get the full record of a single note (including hashtags), use mode note:

actor.call(run_input={"mode": "note", "notes": ["6aa72fd3000000002802c3f4"]})

xhslink.com share links work too, so you can paste links straight from the app.

What it costs

The Actor is pay-per-event:

Event	Price
Run start	$0.01 per run
Note from a list (search, creator notes) or a comment	$5 per 1,000
Note detail (mode `note`)	$20 per 1,000
Creator profile	$20 per 1,000

Check the store page for the current price on your plan. Search results are the cheapest way to collect notes. Use note mode only when you need the full record, such as hashtags.

Going further

Schedule it. Create a saved task with your keyword and run it daily from the Apify Console to track a trend over time.
No-code exports. Every run's dataset downloads as JSON, CSV or Excel, and it connects to Google Sheets, Zapier and Make.
AI agents. The Actor is available through the Apify MCP server, so an agent can call it directly.
Other platforms. The same approach works for Weibo (including the trending list) and Bilibili.

The data is public content only. You are responsible for using it in line with applicable laws and Xiaohongshu's terms.

If a field you need is missing, open an issue on the Actor page. I read them.