E-mail Batch Sender Suggestions Needed

[tenetarns]

Hi there - I run a website for bands and musicians that, as one of the features, allows them to gather mailing list subscriptions and then send out a newsletter to all their subscribers. Just so you know what I'm using, the site is built in ColdFusion, uses an Access database, and uses the CFMAIL tag to process the sending.

The problem is this: Several of the bands on there are getting some pretty hefty mailing lists, with thounsands of people on there. In my original version of the site, when they would send the list, it would basically bomb my hosts e-mail server all at once with thousands of outbound e-mails.

Seeing that some e-mails were getting dropped and that there were some problems, I created a temporary arrangement. Basically, the site now sends those thousands of e-mails in 200-email batches. It takes time, but it works. The number of dropped e-mails has nearly disappeared.

However, this is NOT a final solution for this, and I'm trying to figure out a new way to approach this. Are there any mail server services that can take thousand e-mail drops? Are there any kind of special CF tags or coding methods that would help?

I'm pretty much against a wall here, and I'm trying to figure out a way for this to work as best as possible. I want to keep my clients happy.

Thanks for any suggestions you might have!

[Musicman]

Him

rumour has it that unix servers can deal better with large maildrops ... but they would not run access :)

You probably cannot convince your host to change their email software, so why not improve your current scheme...
You are probably uploading the actual mail message to send. Now when you start, the server could send the first hundred or so mails and return a html page. After a minute the html page refreshes and asks for sending the next lot.
I once did that as either php or cgi; I am not sure whether I still have that code lying around

Musicman

[tenetarns]

Thanks - actually, that's the exact method I'm using, as I described. It sends the first batch of 200, waits a minute, sends the next 200, etc.

However, I've got some bands on there that have over 3000 subscribers. They basically have to wait around for over 15 minutes while it sends the batches.

I'm wondering if there's a better method.

[frostman]

if you were running on *nix, you might still have this problem (though at higher threshholds), and i might recommend the following solution. however, since you're running on windows, i don't really know how applicable this is (i don't have much server experience with windows).

so, in *nix i would do this:
when a user (a band) clicks the "Spam My Fans" button, just make a note of it on the server (in the database). then have a cron job (a process that runs every x minutes in the background) check for those notes and, when it finds one, put the message and all the e-mail addresses of the fans into a queue (also in the DB if you like).

then that cron job or another is always checking this queue, and whenever there is something queued, grabbing N number of addresses and sending to them (and deleting them from the queue, of course).

the rationale is that the users (bands) don't need the 3000+ messages sent *all at once.* if it takes ten minutes, it's not a problem... it's just a problem if they have to stare at the monitor for those ten minutes.


i'm sure you can do the same with windows, there must be some cron-equivalent timed background program thingy.


cheers

[tenetarns]

i'm sure you can do the same with windows, there must be some cron-equivalent timed background program thingy.

I'm sure there is, too. The thing is, though, that I don't have an on-site server. I rely on a company located in Canada for the hosting and they're simply not going to let me toy with their servers. :(

Additionally, of course, I've simply never done anything like that before. I'd like to find out how to do it.

[1a2b3c4d]

You could use a commercial service.

We use Constant Contact. We have clients who send as little as 1,000 emails and some who send 40,000. The service gives me fantastic reporting on clickthroughs, opened emails, bounces, etc.

You can try it for free to see if it works for you,

http://reseller.roving.com/index.jsp?pn=visionarywebworks&pd=visionarywebworks

[frostman]

Maybe you should switch to some kind of Unix/Linux/BSD/Etc based service.

I know that's a big pain, but you'll get to use much more standard tools, and it's not very expensive anymore. Cron and mySQL (and php perl ruby whatever) are standard equipment in the Unix hosting world. Anybody offering less, unless it's as-good-as-free, is cheating.

Some folks will also let you set up real databases, like PostgreSQL, with which you can learn lots of stuff that's applicable to the commercial world (I'm a database guy btw). And anyway unless you are using SQLserver you'll probably find mySQL a big improvement, at least performance-wise.

I personally have two hosted accounts and two servers, one a colo and one an experimental server on a DSL line. Depending on where you live, you might like to try the DSL solution- speakeasy.net are the people to talk to. If you are going to have premium DSL anyway it's nice to also have a server.

Just as a reference point, one of my providers, http://www.cwihosting.com/ has good price/performance for the kind of thing you're talking about.

If you do switch providers, while you're shopping around be sure to let them know you plan to send a lot of mails and in what context. Things being what they are today, you might get preemptively shut down if too much mail goes out from your account, as people might suspect you of spamming. Saves a lot of time to explain the situation upfront.

[tenetarns]

It would actually be a tremendous pain to switch platforms. This site has taken me quite a long time to build, since it's what I do in my off-time. It's built entirely in ColdFusion, which doesn't run on Unix.

Another thing is that, well, I'm primarily a web/flash/multimedia/music/video designer who does *some* coding. I've learned a lot in this area, but I am by no means a dedicated programmer/database guru. I have worked with SQL Server in the past and have a fair amount of experience with it, but it's been so long (over a year and a half) since I touched it that I would basically be starting from scratch.

The ideal - VERY ideal - thing for me would be to setup a colocated server running Windows and ColdFusion MX. However, that's extremely expensive for me right now, especially since it's a site that doesn't draw all that much revenue. The lowest plans I've seen for that kind of thing are $400+ a month.

I'd also consider a dedicated DSL line. I have a standard DSL line here, off of which I run my home network. How much do dedicated DSL lines cost normally?

Also, what kind of performance can you get out of a dedicated DSL? How does it compare to a shared or standard T1?

[frostman]

Windows hosting has also gotten a lot more competitive recently. You should do some net searching, you might find someone who will give you all the features you need for a reasonable price.

And $400 for a colo... well, you can get managed hosting cheaper than that, even with Windows. I just started using ServePath for a unix project, and I got the cheapest one (for alpha/beta development) which is something like $90/mo if you prepay for a year (no setup that way). It's not much of a machine but it's very cheap and the bandwidth allocation is quite generous.

Also remember that a colo is only good if you have either plenty of money to pay the techs or, better, physical access to the box.

The DSL option for you would probably make sense to handle the mailing, but not the whole site, since you probably have more traffic than normal DSL would happily feed. The only special things you need are:

1) A fixed IP.
2) A provider who doesn't mind you running a server.

A lot of providers won't let you run a server off your DSL line because their price structure is such that they can only survive if (almost) nobody uses their allotted bandwidth. They're afraid (reasonably) that servers will use all that bandwidth and they'll be hosed.

Anyway, SpeakEasy, if available in your area, has a good reputation in that regard.

You could also try either the DSL route or a very cheap (like $15) hosting package and use it *just* for the email part, leaving the rest of the site where it is.

What's the URL by the way? I used to work in online music (DRM & database stuff) so I'm curious.


cheers

-- f.