Why do I get an OutOfMemoryError when inserting 50,000 objects into HashMap?

I am trying to insert about 50,000 objects (and therefore 50,000 keys) into a java.util.HashMap<java.awt.Point, Segment>. However, I keep getting an OutOfMemory exception. (Segment is my own class - very light weight - one String field, and 3 int fields).

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.resize(HashMap.java:508)
    at java.util.HashMap.addEntry(HashMap.java:799)
    at java.util.HashMap.put(HashMap.java:431)
    at bus.tools.UpdateMap.putSegment(UpdateMap.java:168)

This seems quite ridiculous since I see that there is plenty of memory available on the machine - both in free RAM and HD space for virtual memory.

Is it possible Java is running with some stringent memory requirements? Can I increase these?

Is there some weird limitation with HashMap? Am I going to have to implement my own? Are there any other classes worth looking at?

(I am running Java 5 under OS X 10.5 on an Intel machine with 2GB RAM.)

Asked by: Abigail674 | Posted: 23-01-2022

Answer 1

You can increase the maximum heap size by passing -Xmx128m (where 128 is the number of megabytes) to java. I can't remember the default size, but it strikes me that it was something rather small.

You can programmatically check how much memory is available by using the Runtime class.

// Get current size of heap in bytes
long heapSize = Runtime.getRuntime().totalMemory();

// Get maximum size of heap in bytes. The heap cannot grow beyond this size.
// Any attempt will result in an OutOfMemoryException.
long heapMaxSize = Runtime.getRuntime().maxMemory();

// Get amount of free memory within the heap in bytes. This size will increase
// after garbage collection and decrease as new objects are created.
long heapFreeSize = Runtime.getRuntime().freeMemory();

(Example from Java Developers Almanac)

This is also partially addressed in Frequently Asked Questions About the Java HotSpot VM, and in the Java 6 GC Tuning page.

Answered by: Kate121 | Posted: 24-02-2022

Answer 2

Some people are suggesting changing the parameters of the HashMap to tighten up the memory requirements. I would suggest to measure instead of guessing; it might be something else causing the OOME. In particular, I'd suggest using either NetBeans Profiler or VisualVM (which comes with Java 6, but I see you're stuck with Java 5).

Answered by: Melissa428 | Posted: 24-02-2022

Answer 3

Another thing to try if you know the number of objects beforehand is to use the HashMap(int capacity,double loadfactor) constructor instead of the default no-arg one which uses defaults of (16,0.75). If the number of elements in your HashMap exceeds (capacity * loadfactor) then the underlying array in the HashMap will be resized to the next power of 2 and the table will be rehashed. This array also requires a contiguous area of memory so for example if you're doubling from a 32768 to a 65536 size array you'll need a 256kB chunk of memory free. To avoid the extra allocation and rehashing penalties, just use a larger hash table from the start. It'll also decrease the chance that you won't have a contiguous area of memory large enough to fit the map.

Answered by: Roland269 | Posted: 24-02-2022

Answer 4

The implementations are backed by arrays usually. Arrays are fixed size blocks of memory. The hashmap implementation starts by storing data in one of these arrays at a given capacity, say 100 objects.

If it fills up the array and you keep adding objects the map needs to secretly increase its array size. Since arrays are fixed, it does this by creating an entirely new array, in memory, along with the current array, that is slightly larger. This is referred to as growing the array. Then all the items from the old array are copied into the new array and the old array is dereferenced with the hope it will be garbage collected and the memory freed at some point.

Usually the code that increases the capacity of the map by copying items into a larger array is the cause of such a problem. There are "dumb" implementations and smart ones that use a growth or load factor that determines the size of the new array based on the size of the old array. Some implementations hide these parameters and some do not so you cannot always set them. The problem is that when you cannot set it, it chooses some default load factor, like 2. So the new array is twice the size of the old. Now your supposedly 50k map has a backing array of 100k.

Look to see if you can reduce the load factor down to 0.25 or something. this causes more hash map collisions which hurts performance but you are hitting a memory bottleneck and need to do so.

Use this constructor:

(http://java.sun.com/javase/6/docs/api/java/util/HashMap.html#HashMap(int, float))

Answered by: Walter772 | Posted: 24-02-2022

Answer 5

You probably need to set the flag -Xmx512m or some larger number when starting java. I think 64mb is the default.

Edited to add: After you figure out how much memory your objects are actually using with a profiler, you may want to look into weak references or soft references to make sure you're not accidentally holding some of your memory hostage from the garbage collector when you're no longer using them.

Answered by: Gianna389 | Posted: 24-02-2022

Answer 6

Also might want to take a look at this:


Answered by: Alford680 | Posted: 24-02-2022

Answer 7

Implicit in these answers it that Java has a fixed size for memory and doesn't grow beyond the configured maximum heap size. This is unlike, say, C, where it's constrained only by the machine on which it's being run.

Answered by: Kelvin284 | Posted: 24-02-2022

Answer 8

By default, the JVM uses a limited heap space. The limit is JVM implementation-dependent, and it's not clear what JVM you are using. On OS's other than Windows, a 32-bit Sun JVM on a machine with 2 Gb or more will use a default maximum heap size of 1/4 of the physical memory, or 512 Mb in your case. However, the default for a "client" mode JVM is only 64 Mb maximum heap size, which may be what you've run into. Other vendor's JVM's may select different defaults.

Of course, you can specify the heap limit explicitly with the -Xmx<NN>m option to java, where <NN> is the number of megabytes for the heap.

As a rough guess, your hash table should only be using about 16 Mb, so there must be some other large objects on the heap. If you could use a Comparable key in a TreeMap, that would save some memory.

See "Ergonomics in the 5.0 JVM" for more details.

Answered by: Julian504 | Posted: 24-02-2022

Answer 9

The Java heap space is limited by default, but that still sounds extreme (though how big are your 50000 segments?)

I am suspecting that you have some other problem, like the arrays in the set growing too big because everything gets assigned into the same "slot" (also affects performance, of course). However, that seems unlikely if your points are uniformly distributed.

I'm wondering though why you're using a HashMap rather than a TreeMap? Even though points are two dimensional, you could subclass them with a compare function and then do log(n) lookups.

Answered by: Thomas706 | Posted: 24-02-2022

Answer 10

Random thought: The hash buckets associated with HashMap are not particularly memory efficient. You may want to try out TreeMap as an alternative and see if it still provide sufficient performance.

Answered by: Daryl619 | Posted: 24-02-2022

Similar questions

java - OutOfMemoryError while inserting data from large JSON file into MySQL database

I am reading a JSON file of roughly 6 GB using Java. The goal is to import all the JSON data into a MySQL database. The program runs fine for few initial records but then throws a java.lang.OutOfMemoryError with message Java heap space error. I am using the JsonParser class to read the file and using nextToken() to read the next JSON token. The system on which the program is ...

java - Grails, Inserting lots of data using withTransaction results in OutOfMemoryError

I'm using Grails 1.1 beta2. I need to import a large amount of data into my Grails application. If I repeatedly instantiate a grails domain class and then save it, the performance is unacceptably slow. Take for example importing people from a phone book: for (each person in legacy phone book) { // Construct new Grails domain class from legacy phone book person Person person = new Person(...) per...

memory management - How to handle OutOfMemoryError in Java?

This question already has answers here:

java - Big XML file and OutOfMemoryError

I’m trying to parse a XML file up to 500 mb in java. I tried to use SAX but it gives me this error java.lang.OutOfMemoryError: Java heap space at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append(Unknown Source) Can you help me? Thanks a lot. P.S. Smaller XML files works just fine

java - OutOfMemoryError - why can a waiting Thread not be garbage collected?

This simple sample code demonstrates the problem. I create an ArrayBlockingQueue, and a thread that waits for data on this queue using take(). After the loop is over, in theory both the queue and the thread can be garbage collected, but in practice I soon get an OutOfMemoryError. What is preventing this to be GC'd, and how can this be fixed? /** * Produces out of memo...

java - OutOfMemoryError in j2me application

I am getting OutOfMemoryException in a J2ME Application. How can I find what is causing this error? And how to prevent getting this error ? I make a http request. While the request isn't completed the screen shows a loading image (like a browser shows when a page is loading). This was done by creating an Image and repainting the screen. create image 1 -> repaint -> create image 2-> repa...

java - Console output retained in char[] causes OutOfMemoryError

I've got an applet that keeps running out of memory. A heap dump analysis shows that the culprit is a char[] owned by the traceMsgQueueThread that contains the entire contents of the java console output. This grows over time as log messages are sent to the console until eventually the applet runs out of memory. This only seems to occur for a handful of clients, and has been seen on various JREs from pre 1.5 to 1....

java - Tomcat Session Eviction to Avoid OutOfMemoryError

We are running a vendor-supplied webapp in Tomcat 5.5 using the StandardManager for sessions (in memory). As sessions can get quite large (20M+), running out of heap space is a serious concern. Users want to keep sessions around for a couple of hours if possible but would rather evict sessions than run out of heap space. It does not appear that the vendor properly implemented Serializable in session'ed objects, so switc...

multithreading - OutOfMemoryError in a separate Java thread

Consider that I have a main Thread which executes a new Runnable in a new Thread. Now, while the new Thread is executing, the Java VM runs out of memory and throws an OutOfMemoryError. What happens? Does the target thread stop? Will the main thread continue? When the new Thread crashes, will the VM reclaim the memory from it and let execution continue?

memory - Java OutOfMemoryError message changes when trying to create Arrays of different sizes

In the question by DKSRathore How to simulate the Out Of memory : Requested array size exceeds VM limit some odd behavior was noted when creating an arrays. When creating an array of size Integer.MAX_VALUE an exception with the error java.lang.OutOfMemoryError Requested ar...

java - OutOfMemoryError

I have one main class that contains 5 buttons each link to a program/package. Each package runs a jmf program that capture images from a webcam and it also loads about 15 images from file. The 1st program to load(regardless of which button i press) always runs correctly. But When i run a program after the 1st program ends, java.lang.OutOfMemoryError: java heap space occurs. Im not sure i...

Still can't find your answer? Check out these amazing Java communities for help...

Java Reddit Community | Java Help Reddit Community | Dev.to Java Community | Java Discord | Java Programmers (Facebook) | Java developers (Facebook)