dcsimg
A Flash Developer Resource Site

Page 1 of 3 123 LastLast
Results 1 to 20 of 45

Thread: Embedding large text file

  1. #1
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269

    Embedding large text file

    Hey Chaps,

    I'm sure there's an easy solution to this, but I haven't figured it out yet!

    I have a very large text file (it's a 270,000 word list with one word per line) which I need to bring into my AS3 code. Importing it via url works just fine - loading it in as a string then converting to an array - no problems.

    However, I need to embed it into my code somehow so that there are no external files (for easy distribution). Using a search and replace tool, I've tried several things, none of which have worked including:

    Converting to a single space delimited string:

    Code:
    var wordList:String = "word1 word2 word3 ...";
    Result: crashes compiler.

    Code:
    Converting to an array: var wordList:Array = ["word1","word2","word3", ...];
    Result: crashes compiler too.

    The same tests work fine with much smaller sized word lists, but apparently the size of my word list is too large to work. I know flash can handle strings and arrays of this size, because as I said, it works fine when importing the text file via url. I just can't seem to get it to compile it when I embed it into code.

    Any ideas? Thanks
    Last edited by _Ric_; 04-01-2009 at 10:49 AM.

  2. #2
    Señor Member Mavrisa's Avatar
    Join Date
    Oct 2005
    Location
    Canada
    Posts
    506
    IF you're using cs4 and you have a copy of flex (the sdk works as well I think) you can embed it using the embed tag right in your actionscript code. This basically tells the compiler to include the file when making the swf.
    Haikus are easy
    But sometimes they don't make sense
    Refrigerator

  3. #3
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    That sounds exactly what I need - but unfortunately I don't have cs4 or flex - just cs3.

  4. #4
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Still struggling with this - I don't seem to be getting anywhere! I've even tried pasting the entire wordlist into a dynamic textfield in the library and referencing that from code. Apart from slowing Flash down to an almost standstill, the string I managed to extract from it only contained about 6000 of the 270000 words which were meant to be there. I'm Running out of ideas ... anyone?
    Last edited by _Ric_; 04-01-2009 at 12:07 PM.

  5. #5
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Still not figured this out. I've now also tried this:

    Code:
    function createList():{
    
    wordList:Array = new Array();
    
    wordList.push("word1");
    wordList.push("word2");
    wordList.push("word3");
    wordList.push("word4");
    .....
    
    }
    Using a search and replace tool, I tried this for each word in the word list, resulting in an actionscript file just over 270,000 lines long. Result: crashes the compiler - again!

    So I tried reducing the length of the list, and discovered that the same thing works for a list of 50,000 words, but not much more. So I thought let's just split the list into 5 or 6 separate include files, each contained in a separate function, and call the functions in order:

    Code:
    include "list1.as";
    include "list2.as";
    ...
    
    
    createList1();  //first 50,000 words in 'list1.as'
    createList2();  //second 50,000 words in 'list2.as'
    ....
    That also crashes the compiler. Any one of the 50,000 word lists by themselves will compile, but it won't compile more than one of them - and I need 5 or 6 of them at this size.

    I've never had a need for creating my own classes before, so I'm not really sure how to do that yet, but does anyone think it would be more likely to work that way? ie. instead of including a large script, importing it?

  6. #6
    AS3 Mod
    Join Date
    Sep 2007
    Location
    O-H-I-O
    Posts
    2,385
    I don't know if this would be an option but you could try storing your word list in an XML file and using the following code to parse though it:

    PHP Code:
    var myArray:Array = new Array();
    var 
    wordsXML:XML = new XML();
    wordsXML.ignoreWhite true;

    wordsXML.onLoad = function(s:Boolean):Void {
        if(
    s) {
            var 
    node this.firstChild;
            var 
    0len node.childNodes.length;
            
            for(; 
    leni++) {
                
    myArray[i] = (node.childNodes[i].childNodes[0]);
            }
        }else{
            
    trace('Failed to load xml');
        }
    }

    wordsXML.load('words.xml'); 
    PHP Code:
    <?xml version="1.0" encoding="utf-8"?>
    <words>
        <word>asdfasdf</word>
        <word>asdfasdf</word>
        <word>asdfasdf</word>
        <word>asdfasdf</word>
        <word>asdfasdf</word>
    </words>
    I think the ideal solution would be to store your word list in a database and use PHP to communicate with your flash project. Then just query that word list when you need to. This would be a TON faster and much more reliable than what you're doing.

  7. #7
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Hey sstalder - thanks for the ideas. Unfortunately storing the list externally isn't an option - it has to be embedded into the swf. This would definitely be easier if I could do that!

    I did think about xml. I think the main problem is that the word list is already 3MB, and adding all those xml tags would inflate it even more. I will try it though as I've tried just about everything else I can think of!

    By the way - is that AS2? You don't happen to have the same thing in AS3 do you?


  8. #8
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Ok - well I've tried wrapping the word list into an XML object, and putting it into an include file. Same problem as before - works if the list is relatively small, but crashes the compiler with the full list. I'm coming to the conclusion that the problem I'm having isn't to do with how I store the list - string/array/xml or whatever - I think the problem is including an actionscript file of this size. I wasn't aware there was a limit on the size of include files, but I seem to have reached it ... even if it's split into smaller include files. Anyone come across this before - or even better, a solution to it?


    <edit> Uh oh ... looks like my suspiscions may have been right - according to this page at Adobe, there is a 32KB limit on any actionscript file. It doesn't say what the maximum of ALL actionscript code in one swf is, but from googling around, it seems there IS a limit, and I guess I'm going way over it.

    Right ... so what now?!! I think I may need to get creative. Anyone know anything about encoding data using bitmaps, or similar?
    Last edited by _Ric_; 04-02-2009 at 12:39 PM.

  9. #9
    AS3 Mod
    Join Date
    Sep 2007
    Location
    O-H-I-O
    Posts
    2,385
    I think the challenge here is you are trying to use flash for something it was not made for (a database). Will this only be ran locally or is there any certain reason that you can't use a database of any sort?

    Aside from that I would probably start writing some sort of an engine that does whatever you need to do within flash. Maybe a starting point would be to start building out a library of .as files:

    A.as (all words that start with 'a')
    B.as (all words that start with 'b')
    C.as (all words that start with 'c')

    And so on, assuming this lets you stay within the size limitations. Then you would write a small class that handles all your parsing through those objects, etc. That would probably be my approach but without knowing exactly what this is for I can't offer much better suggestions

  10. #10
    Will moderate for beer
    Join Date
    Apr 2007
    Location
    Austin, TX
    Posts
    6,801
    I would suggest downloading and using the free flex sdk to embed a compressed version of the file. You can either do your whole project in that, or produce a swf to put in the library for your cs3 project.

    If you want to avoid the flex sdk entirely, here's a way.

    1. load data externally, as currently done.
    2. write array of words into bytearray
    3. call compress on that bytearray
    4. use that bytearray as the source of a bitmapdata
    5. encode that bitmapdata as a png.
    6. export the png
    7. embed the png as an asset in your project and use the reverse process to get your words.

  11. #11
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Thanks sstalder - I've also tried that (splitting the list into separate .as files), but that still breaks the compiler once the total number of lines in all scripts goes much over 50,000 lines. I've now seen that other people have had the same problem. As you say - Flash wasn't designed for this.

    5tons: Could I do that without Flex anyway, by breaking the word list into smaller parts, compiling into maybe 5 or 6 swf's, then importing them all into the library? How would I access the data once they are embedded into an imported swf?

    The bitmap data method also looks like a good idea, which I will try if embedding swf's doesn't work out.

    Thanks.

  12. #12
    Will moderate for beer
    Join Date
    Apr 2007
    Location
    Austin, TX
    Posts
    6,801
    Yes, you could probably do it by making piecemeal swfs and importing those into the library. Each swf would have to have a public method or property which you could call to get the data you need. Have them all implement an interface or extend a common class so you can cast correctly to call that method.

    After writing that previous post up, I was considering writing a utility to automate the process and putting it on my website. Do you think it's worth it? Basically, you'd upload a file and get back a png. I'd then provide a class to retrieve the original contents as a ByteArray. After that, you'd be on your own to interpret the ByteArray.

  13. #13
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Well I'd certainly use a utility like that - I know it would help with dealing with large amounts of data, but I guess it might also be useful for data encryption.

    For the time being though I'll have a go at the multiple comiled swf's approach. Thanks for the advice.

  14. #14
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Ok - after much experimenting, I managed to precompile the word list into several swc components - each containing a chunk of the word list, and brought them in to the library of my main project. I tried it with one of these components containing about 20,000 lines of code - fine. Two of them - fine. Three of them ..... crashes the compiler ('unknown error generating byte code').

    So it seems pretty clear now that the problem is that Flash just can't handle this much code, even if it's precompiled and imported. In fact, the amount of code I've managed to compile seems about the same whether it's precompiled components, or just included .AS files.

    I've now read a few more reports of other people finding the same problem once the amount of code goes over a certain limit.

    I'm going to try the png encoding idea now. I'll start a new thread for that when I get stuck.

  15. #15
    Will moderate for beer
    Join Date
    Apr 2007
    Location
    Austin, TX
    Posts
    6,801
    And with that, you've pushed me to try to make the utility. Race ya. I won't be able to start until about 8:30pm central time, so enjoy your head start.

  16. #16
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    Haha - let's go! It'll be very interesting for me to see how quick your version is at coverting back to data, and whether it can handle the amount of data I need to throw at it. I can't really get going on this properly until Monday, so you have the head start!

  17. #17
    Will moderate for beer
    Join Date
    Apr 2007
    Location
    Austin, TX
    Posts
    6,801
    Out of curiosity, how much data do you need to throw at it? The png method does have a limit, due to bitmapdata size constraints.

    If you could zip up and send me sample data, that'd be awesome.

  18. #18
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    It's the 'sowpods' word list - about 270000 words, which is basically the international list of allowed scrabble words - the zipped file is freely available on the internet scrabble club here: http://www.isc.ro/lists/sowpods.zip

    I wouldn't be surprised if it had to be broken down to make several images - I did a very rough calculation and think as a single image it might be something like 2000x2000 pixels

  19. #19
    Will moderate for beer
    Join Date
    Apr 2007
    Location
    Austin, TX
    Posts
    6,801
    Actually, that file is only 702785 bytes, which would only require around 420x420 pixels, but brings up an interesting issue with padding. It does not divide evenly by 4, so the last pixel is going to have to be padded with 0s or something. Math.ceil(702785/4) = 175697, which does factor into two numbers acceptable for a bitmap (7639,23), but in general we can't rely on that since the max dimension is 8191, we want to get it sized as close to square as we can, which requires further padding. 420*419 = 175980, which is just big enough. We could pad the bytearray with enough bytes to fill it out then write out our png encoded bitmap, but to get it back, we have to know how much padding there was. The logical place to put that is in the png metadata, but I don't know if the PNGEncoder supports that. I'll have to look around.

  20. #20
    Senior Member
    Join Date
    Jan 2008
    Location
    UK
    Posts
    269
    I think its more unzipped though - about 3 mb.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  




Click Here to Expand Forum to Full Width

HTML5 Development Center