This is an updated code for simulating user-generated email traffic, for the purpose of network simulation. (previous post)

It is based on the analysis of hundreds of gigabytes of mails (see this post).

This samples email size. You give this program a traffic rate, and it tells you when a user sends a mail of what size (two-column list output). It is based on the distribution of 402158 sent-emails, made available by Phill Macey (thank you!). Here you can see the distribution, and the sampler:

As you can see, the generator (by-table) follows the empirical distribution very well. The sampler is very fast (1700000 lines per second on my machine).

example usage: Generate five mails with 100 bytes/second throughput

$ gcc -pg -lgsl -lm -O3 -ansi -pedantic -Wall -Werror -Wextra emailgen-phill-macey.table.c -o emailgen-phill-macey.table.exe
$ GSL_RNG_SEED=12144 ./emailgen-phill-macey.table.exe 5 100 
0	4417
44	1262
56	791
63	829
71	1137

Which means, at second 44 the user wants to write a mail of 1262 bytes.

The full code, emailgen-phill-macey.table.c:

#include<stdio.h>
#include<stdlib.h>
#include<assert.h>
#include<math.h>
#include<gsl/gsl_rng.h>
 
gsl_rng * rng;
void init_rand() {
	gsl_rng_env_setup();
	rng = gsl_rng_alloc(gsl_rng_default);
}
 
 
int get_random(int min, int max) {
	unsigned int umax;
	assert(max > min);
	umax = max - min - 1;
	return gsl_rng_uniform_int (rng, umax) + min;
}
 
double probabilities[15] = {
	0.21380651385774, /* <= 1024 = pow(2,0+10)*/
	0.49594935323927, /* <= 2048 = pow(2,1+11)*/
	0.70676450549287,
	0.81264080286852,
	0.86918076974721,
	0.89862442124737,
	0.92154576062145,
	0.94019514718096,
	0.95773303030152,
	0.97112577643613,
	0.98311111553171,
	0.99845085762312,
	1
};
 
double interpolate(double yleft, double yright, double x, double xleft, double xright) {
	double deltax = xright - xleft;
	double deltay = yright - yleft;
	double k = deltay/deltax;
	return (x - xleft) * k + yleft;
}
 
unsigned long getnextsize() {
	double prob = gsl_rng_uniform_pos(rng);
	int i = 0;
	double j;
	unsigned long hardlowerlimit = 600;
	/*unsigned long hardupperlimit = 10*1024*1024;*/
	unsigned long size;
 
	while(probabilities[i] <= prob) {
		i++;
	}
	if (i>0) {
		j = interpolate(i-1, i, prob, probabilities[i-1], probabilities[i]);
		size = pow(2, j + 10);
		if(size < hardlowerlimit) {
			return getnextsize();
		}
	}else{
		size = get_random(hardlowerlimit, pow(2, 0 + 10));
	}
 
	return size;
}
void usage(char * progname) {
	fprintf(stderr, "%s: SYNAPSIS: number_of_mails throughput \n"
		"\n"
		"\tnumber_of_mails\thow many mails should be generated\n"
		"\tthroughput\tbytes per (virtual) second to send\n"
		"\n"
		"This program prints a 2-column list of time (in seconds) and\n"
		"size (in bytes) to simulate a email traffic. \n"
		"Configure your network simulator to transmit this number of bytes\n"
		"at that time. (This program does not send mails.)\n"
		"\n"
		"(c) Johannes Buchner\n", progname);
	exit(0);
}
 
int main(int argc, char ** argv) {
	int n;
	double throughput;
	unsigned long bytes_sent = 0;
	unsigned long nextsize;
	unsigned long time = 0;
 
	if (argc != 3) {
		usage(argv[0]);
	}
	init_rand();
	throughput = atof(argv[2]);
	n = atoi(argv[1]);
	if (throughput < 0 || n < 0) {
		usage(argv[0]);
	}
 
	while (n-- > 0) {
		nextsize = getnextsize();
		printf("%lu\t%lu\n", time, nextsize);
		bytes_sent += nextsize;
		time += nextsize /* bytes */ / throughput /* bytes per sec */;
	}
	return 0;
}

Following up from a previous post and a post to the qmail and postfix mailing lists:

I analysed the distribution of email sizes. I wasn’t interested in spam, automated emails or mailing lists, rather user-written mails.

I analyzed the sizes of mails in user inboxes. I am always looking at buckets (how many mails are smaller than size
x, but bigger than the last buckets).  I used the following upper limits: 1024
1536 2048 2560 3072 4096 5120 6144 7168 8192 10240 12288 14336
16384 20480 24576 28672 32768 40960 49152 53248 57344 65536 131072
262144 524288 1048576 10264576 and 1073741824 (bytes).

I laid out some interpretations about my inbox as preliminary insight. Now I want to publish the results  from the friendly people that answered my call for data.

Ordered by cardinality (number of mails in dataset):

mymail                             5069
thomas.schwinge_priv               8759
linux.kernel                      11127
charles_cazabon-nospam            22555
phoemix_harmlesshu                30991
thomas.schwinge_tech_mailinglist 154508
phill_macey_sent                 402158
spamsizes_steff                 1044713
michaelreck                     4086621 (700GB, ~40000 users)
phill_macey_inbox               7999737
phill_macey_all                 8401951 (~300GB)

Thanks go out to Thomas Schwinge, Charles Cazabon, phoemix, Michael Reck from brauchmer.net, Markus Stumpf and Phill Macey.

The empirical distribution functions follow (with downloadable eps versions). X-axis (abscissa) is always size in Bytes, the y-axis (ordinate) is the portion of mails found with exactly this size. This is calculated by using the number of mails in a buckets above, dividing by the bucket size (difference to lower border) and dividing by the sum of mails.

The upper is single-log, the lower is double-log.

And the cumulative distribution. This is interesting because a shift can not be seen well in the probability functions. These are linear plots: ordinate being cumulative percentage (what percentage of mail is below size x). The thicker the lines, the more data in this dataset.

And logarithmic in size:

Here are the plots in better quality (as eps): overview-density overview-cum overview-cum-log

Warnings:

There are several errors to consider before taking this too literal: (1) It is uncertain how clean the data of large datasets is (spam, mailing-lists, generated data all mixed up).  (2) headers are different on receiving than on sending. (3) What users send is not the same traffic as they receive (automated mails).

If you just want to take one number out of this, the median size of a mail is around 30kB (watch the 50% line).

For my purposes of modeling a human generating mails, this is fantastic (dataset phill_macey_sent). This is covered in my next post.

Comments are always welcome!

In network simulation, traffic has to be simulated. Here I’m looking at email traffic. Often, CBR (constant bit rate) is used.

For a more realistic email generator I wanted to find out how the emails are distributed in size.

The only post that goes into detail is found at

http://osdir.com/ml/freebsd.devel.net/2002-10/msg00203.html

The resolution of this is quite poor though, especially between 4KB and 0 would be interesting.
However, below is a sampler that generates random email sizes, distributed such that it conforms to the distribution.
Between 2048 and 900 (lower limit, headers almost always need that much space) a uniform distribution is used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

/* cumulative probabilities */
double probabilities[12] = {
	0.2353, /* < 2048 = pow(2, 0+11) */
	0.4917, /* < 3096 = pow(2, 1+11) */
	0.676,
	0.7958,
	0.8536,
	0.9047,
	0.933,
	0.9574,
	0.9724,
	0.9849,
	0.9996,
	1.0
};
 
double interpolate(double yleft, double yright, double x, double xleft, double xright) {
	double deltax = xright - xleft;
	double deltay = yright - yleft;
	double k = deltay/deltax;
	return (x - xleft) * k + yleft;
}
 
unsigned long getnextsize() {
	double prob = gsl_rng_uniform_pos(rng);
	int i = 0;
	double j;
	unsigned long hardlowerlimit = 900;
	/*unsigned long hardupperlimit = 10*1024*1024;*/
	unsigned long size;
 
	while(probabilities[i] <= prob) {
		i++;
	}
	assert(probabilities[i] > prob);
	if (i > 0) {
		j = interpolate(i-1, i, prob, probabilities[i-1], probabilities[i]);
		size = pow(2, j + 11);
		if(size < hardlowerlimit) {
			return getnextsize();
		}
	}else{
		size = get_random(hardlowerlimit, pow(2, 0+11));
	}
 
	return size;
}

The measured resulting cumulative distribution function, as well as a comparison to my own mailbox (which might be atypical, who knows).

It would be interesting to get more detailed and newer histograms.
I also considered modelling it using a Maxwell–Boltzmann distribution, but generating that seemed more complex and on closer look, the form does not really fit.

Full sampler code:
emailgen.table.c