guppy/heapy - Profile Memory Usage in Python¶
The guppy is a python package that provides the user status of the current heap and objects present in the heap. This information can help the developer with memory profiling and analysis. The guppy has a module named heapy which provides a list of important methods and current heap status. The guppy when called for heap status gives us a special C-nodesets object which has heap status for each object present in the memory. It has information about all reachable and/or unreachable objects present in the heap as a special type of list. It also provides information about a number of objects, % of memory occupied by the object in memory, size in bytes, and type information as well. Apart from this guppy module provides a list of useful methods that can let us access an individual element of the heap status, find the difference between heap statuses, find the size of objects in bytes, and many other functionalities. We'll be explaining the usage of guppy to collect information about memory usage in python through various examples as a part of this tutorial.
Example 1¶
We'll first explain how to access heap status as a part of our first example and then will slowly build on it.
guppy.hpy() - The guppy has method named
hpy()which gives us access toheapyobject. This will be the first method called by all our examples in order to get theheapyobject which will provide a common interface for accessing heap status and performing other operations as well.heap() - This method is available through object created by calling
hpy()method. This method provides a list of objects accessible from the root of the heap which are reachable and visible. The returned result is presented in a table format. Please make a note that it does not include objects used as a part of guppy.setref() - This method is available through heapy object created by calling
hpy(). It is used to set a reference point and all objects created after this reference point will be available in the next call to theheap()method for heap status. It'll not include all objects present in heap but only once created after the reference point.
Below we have explained the usage of all these 3 methods. We have first collected heap status at the beginning of the script. We have then set the reference point and retrieved the heap status again. Then we have created a few objects like list, string, and numpy array of random numbers. After the creation of these objects, we have again called the heap() method to get heap status which has information about these objects.
We can see from the output that it has information about a number of objects and the total size of the whole heap as well as object count, size, % of memory used by that object type, and type information. The second heap status does not much info as not much has happened after setting up a reference point. The third heap status object has information about objects created after setting a reference point.
CODE
import guppy
from guppy import hpy
import numpy as np
heap = hpy()
print("Heap Status At Starting : ")
heap_status1 = heap.heap()
print("Heap Size : ", heap_status1.size, " bytes\n")
print(heap_status1)
heap.setref()
print("\nHeap Status After Setting Reference Point : ")
heap_status2 = heap.heap()
print("Heap Size : ", heap_status2.size, " bytes\n")
print(heap_status2)
a = [i for i in range(1000)]
b = "A"
c = np.random.randint(1,100, (1000,))
print("\nHeap Status After Creating Few Objects : ")
heap_status3 = heap.heap()
print("Heap Size : ", heap_status3.size, " bytes\n")
print(heap_status3)
print("\nMemory Usage After Creation Of Objects : ", heap_status3.size - heap_status2.size, " bytes")
OUTPUT
Heap Status At Starting :
Heap Size : 12985791 bytes
Partition of a set of 98239 objects. Total size = 12985791 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 27441 28 4200501 32 4200501 32 str
1 26394 27 2002360 15 6202861 48 tuple
2 13607 14 1021370 8 7224231 56 bytes
3 6772 7 978824 8 8203055 63 types.CodeType
4 6235 6 847960 7 9051015 70 function
5 940 1 801552 6 9852567 76 type
6 273 0 541800 4 10394367 80 dict of module
7 940 1 509240 4 10903607 84 dict of type
8 1202 1 482112 4 11385719 88 dict (no owner)
9 791 1 191880 1 11577599 89 dict of function
<248 more rows. Type e.g. '_.more' to view.>
Heap Status After Setting Reference Point :
Heap Size : 416 bytes
Partition of a set of 1 object. Total size = 416 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 416 100 416 100 types.FrameType
Heap Status After Creating Few Objects :
Heap Size : 40968 bytes
Partition of a set of 838 objects. Total size = 40968 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 834 100 23352 57 23352 57 int
1 1 0 9024 22 32376 79 list
2 1 0 8096 20 40472 99 numpy.ndarray
3 1 0 416 1 40888 100 types.FrameType
4 1 0 80 0 40968 100 builtins.weakref
Memory Usage After Creation Of Objects : 40552 bytes
Example 2¶
As a part of the second example, we'll explain how to access objects which are not reachable from the root of the heap. This is generally created using cycle references.
- heapu() - This method is available through an object created by calling
hpy()method of guppy module. This method provides us heap status about the list of objects which are not reachable from the root of the heap. The object returned byheapu()method is a status object (guppy.heapy.Part.Stat) which has stats about unreachable objects.
Below we have first retrieved a list of unreachable objects from the heap and then used various methods to access an individual row of status to retrieve information about the individual data type.
If you are interested in learning about memory management in python then please feel free to check our tutorial on the same.
CODE
import guppy
from guppy import hpy
heap = hpy()
print("GC Collectable Objects Which Are not Reachable from Root of Heap")
stats = heap.heapu()
print("Total Objects : ", stats.count)
print("Total Size : ", stats.size, " Bytes")
print("Number of Entries : ", stats.numrows)
print("Entries : ")
print("Index Count Size Cumulative Size Object Name")
for row in stats.get_rows():
print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))
print("\nFirst 5 Entries : ")
print("Index Count Size Cumulative Size Object Name")
for row in stats.rows[:5]:
print("%5d%5d%8d%8d%30s"%(row.index, row.count, row.size, row.cumulsize, row.name))
print("\nDirectly Printing Results Without Iteration")
print(heap.heapu(stat=0))
print("\nMeasuring Unreachable Objects From This Reference Point Onwards")
heap.setref()
print(heap.heapu(stat=0))
OUTPUT
GC Collectable Objects Which Are not Reachable from Root of Heap
Total Objects : 681
Total Size : 119273 Bytes
Number of Entries : 16
Entries :
Index Count Size Cumulative Size Object Name
0 65 50136 50136 dict
1 44 17600 67736 type
2 44 10880 78616 set
3 125 10000 88616 types.WrapperDescriptorType
4 102 8160 96776 builtins.weakref
5 90 6480 103256 types.BuiltinMethodType
6 88 5368 108624 tuple
7 50 3600 112224 types.MemberDescriptorType
8 46 3312 115536 types.MethodDescriptorType
9 13 1737 117273 str
10 3 1176 118449 list
11 7 504 118953 types.GetSetDescriptorType
12 1 168 119121 sys.flags
13 1 80 119201 module
14 1 56 119257 staticmethod
15 1 16 119273 Token.MISSING
First 5 Entries :
Index Count Size Cumulative Size Object Name
0 65 50136 50136 dict
1 44 17600 67736 type
2 44 10880 78616 set
3 125 10000 88616 types.WrapperDescriptorType
4 102 8160 96776 builtins.weakref
Directly Printing Results Without Iteration
Partition of a set of 790 objects. Total size = 137320 bytes.
Index Count % Size % Cumulative % Type
0 77 10 58552 43 58552 43 dict
1 52 7 20800 15 79352 58 type
2 143 18 11440 8 90792 66 types.WrapperDescriptorType
3 44 6 10880 8 101672 74 set
4 102 13 8160 6 109832 80 builtins.weakref
5 108 14 7776 6 117608 86 types.BuiltinMethodType
6 109 14 6712 5 124320 91 tuple
7 69 9 4968 4 129288 94 types.MethodDescriptorType
8 51 6 3672 3 132960 97 types.MemberDescriptorType
9 19 2 2216 2 135176 98 str
<6 more rows. Type e.g. '_.more' to view.>
Measuring Unreachable Objects From This Reference Point Onwards
Partition of a set of 790 objects. Total size = 137320 bytes.
Index Count % Size % Cumulative % Type
0 77 10 58552 43 58552 43 dict
1 52 7 20800 15 79352 58 type
2 143 18 11440 8 90792 66 types.WrapperDescriptorType
3 44 6 10880 8 101672 74 set
4 102 13 8160 6 109832 80 builtins.weakref
5 108 14 7776 6 117608 86 types.BuiltinMethodType
6 109 14 6712 5 124320 91 tuple
7 69 9 4968 4 129288 94 types.MethodDescriptorType
8 51 6 3672 3 132960 97 types.MemberDescriptorType
9 19 2 2216 2 135176 98 str
<6 more rows. Type e.g. '_.more' to view.>
Example 3¶
As a part of this example, we'll explain how we can retrieve an individual entry from the whole heap status object.
Below we have retrieved heap status after creating few lists. The individual object can also be accessed from the heap status object by simply using list indexing. Below we have printed various individual entry of heap status. We have also created a simple method that takes as input size in bytes and returns size in KB/MB/GB.
CODE
import guppy
from guppy import hpy
import numpy as np
heap = hpy()
print("Heap Status At Starting : ")
l1 = [i*i for i in range(1000)]
l2 = np.random.randint(1,100, (1000,1000))
heap_status1 = heap.heap()
print(heap_status1)
print("\nAccessing Individual Element of Heap")
print("\nFirst Element : ")
print(heap_status1[0])
print("\nSecond Element : ")
print(heap_status1[1])
print("\nThird Element : ")
print(heap_status1.parts[2])
def convert_size(size):
if size <1024:
return size
elif (size >= 1024) and (size < (1024 * 1024)):
return "%.2f KB"%(size/1024)
elif (size >= (1024*1024)) and (size < (1024*1024*1024)):
return "%.2f MB"%(size/(1024*1024))
else:
return "%.2f GB"%(size/(1024*1024*1024))
print("\nTotal Heap Size : ", convert_size(heap_status1.size), "\n")
for i in range(10):
print("Size Of Object : %d - "%i, convert_size(heap_status1[i].size))
OUTPUT
Heap Status At Starting :
Partition of a set of 99240 objects. Total size = 21023362 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 69 0 8006950 38 8006950 38 numpy.ndarray
1 27445 28 4200686 20 12207636 58 str
2 26396 27 2002552 10 14210188 68 tuple
3 13609 14 1021552 5 15231740 72 bytes
4 6773 7 978968 5 16210708 77 types.CodeType
5 6235 6 847960 4 17058668 81 function
6 940 1 801552 4 17860220 85 type
7 273 0 541800 3 18402020 88 dict of module
8 940 1 509240 2 18911260 90 dict of type
9 1202 1 482112 2 19393372 92 dict (no owner)
<248 more rows. Type e.g. '_.more' to view.>
Accessing Individual Element of Heap
First Element :
Partition of a set of 69 objects. Total size = 8006950 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 69 100 8006950 100 8006950 100 numpy.ndarray
Second Element :
Partition of a set of 27445 objects. Total size = 4200686 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
1 27445 100 4200686 100 12207636 291 str
Third Element :
Partition of a set of 26396 objects. Total size = 2002552 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
2 26396 100 2002552 100 14210188 710 tuple
Total Heap Size : 20.05 MB
Size Of Object : 0 - 7.64 MB
Size Of Object : 1 - 4.01 MB
Size Of Object : 2 - 1.91 MB
Size Of Object : 3 - 997.61 KB
Size Of Object : 4 - 956.02 KB
Size Of Object : 5 - 828.09 KB
Size Of Object : 6 - 783.16 KB
Size Of Object : 7 - 529.10 KB
Size Of Object : 8 - 497.30 KB
Size Of Object : 9 - 470.81 KB
Example 4¶
As a part of our fourth example, we'll explain how we can find out the difference between two heap status to check how many objects were created between two calls of heap status.
diff() - This method is available as a part of the heap status object which takes as input another heap status to find out objects difference between two statuses. It returns an object of type
guppy.heapy.Part.Statwhich we have explored as a part of our second example.disjoint() - This method is available through a heap status object that takes input another heap status and returns True/False based on whether two heap status is disjoint or not.
Below we have first taken heap status at the beginning. We have then created a few lists of objects and string object. After the creation of these objects, we have again taken another heap status. We have then called the diff() method on the second heap status object passing it first heap status object to get the difference between two screenshots of the heap. We have then looped through stats object and printed the difference of objects.
CODE
import guppy
from guppy import hpy
import numpy as np
heap = hpy()
print("Heap Status At Starting : ")
heap_status1 = heap.heap()
print("Heap Size : ", heap_status1.size, " bytes\n")
print(heap_status1)
a = [i for i in range(1000)]
b = "A"
c = np.random.randint(1,100, (1000,))
print("\nHeap Status After Creating Few Objects : ")
heap_status2 = heap.heap()
print("Heap Size : ", heap_status2.size, " bytes\n")
print(heap_status2)
print("\nMemory Usage After Creation Of Objects : ", heap_status2.size - heap_status1.size, " bytes")
print("\nFinding Out Difference Between Two Heap Status : ")
stats = heap_status2.diff(heap_status1)
print("Whether Two Heap Status Are Disjoint : ", heap_status1.disjoint(heap_status2))
print("Total Objects : ", stats.count)
print("Total Size : ", stats.size, " Bytes")
print("Number of Entries : ", stats.numrows)
print("Entries : ")
print("Index Count Size Cumulative Size Object Name")
for row in stats.get_rows():
print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))
OUTPUT
Heap Status At Starting :
Heap Size : 12986611 bytes
Partition of a set of 98247 objects. Total size = 12986611 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 27449 28 4201065 32 4201065 32 str
1 26394 27 2002488 15 6203553 48 tuple
2 13607 14 1021474 8 7225027 56 bytes
3 6772 7 978824 8 8203851 63 types.CodeType
4 6235 6 847960 7 9051811 70 function
5 940 1 801552 6 9853363 76 type
6 273 0 541800 4 10395163 80 dict of module
7 940 1 509240 4 10904403 84 dict of type
8 1202 1 482112 4 11386515 88 dict (no owner)
9 791 1 191880 1 11578395 89 dict of function
<248 more rows. Type e.g. '_.more' to view.>
Heap Status After Creating Few Objects :
Heap Size : 13027083 bytes
Partition of a set of 99083 objects. Total size = 13027083 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 27449 28 4201065 32 4201065 32 str
1 26394 27 2002488 15 6203553 48 tuple
2 13607 14 1021474 8 7225027 55 bytes
3 6772 7 978824 8 8203851 63 types.CodeType
4 6235 6 847960 7 9051811 69 function
5 940 1 801552 6 9853363 76 type
6 273 0 541800 4 10395163 80 dict of module
7 940 1 509240 4 10904403 84 dict of type
8 1202 1 482112 4 11386515 87 dict (no owner)
9 791 1 191880 1 11578395 89 dict of function
<248 more rows. Type e.g. '_.more' to view.>
Memory Usage After Creation Of Objects : 40472 bytes
Finding Out Difference Between Two Heap Status :
Whether Two Heap Status Are Disjoint : False
Total Objects : 836
Total Size : 40472 Bytes
Number of Entries : 5
Entries :
Index Count Size Cumulative Size Object Name
0 834 23352 23352 int
1 1 9024 32376 list
2 1 8096 40472 numpy.ndarray
3 0 0 40472 ctypes.CFunctionType
4 0 0 40472 numpy.int64
Example 5¶
As a part of this example, we'll explore a few methods available through heap status methods and attributes.
- count - This attribute of heap status object returns an integer specifying a total number of objects present when the status was taken.
- size - This attribute of heap status object returns an integer specifying the total size of the heap when the status was taken.
- referents - This attribute returns another heap status object with only entry for objects which are referred to by other objects.
- referrers - This attribute returns another heap status object with only entry for objects which are referring to other objects.
- stat - It returns
guppy.heapy.Part.Statwhich can be looped to get information about the individual entry of the heap status as we had explained in our previous examples. - dump() - It takes as input filename to which heap status will be dumped.
- load() - It takes as input filename from which heap status will be loaded. This method can be called over the object created by calling the
hpy()method. It returnsguppy.heapy.Part.Statobject of heap status present in the file.
Below we have explained the usage of the above properties and methods.
CODE
import guppy
from guppy import hpy
import numpy as np
heap = hpy()
heap_status1 = heap.heap()
print("\nFew Important Properties/Methods of Heap Status Object(%s) : "%type(heap_status1))
print("\nCount of Objects in Heap : ", heap_status1.count)
print("\nHeap Size : ", heap_status1.size, " bytes")
print("\nHeap 1st Entry : ")
print(heap_status1.parts[0])
print("\nSet of Objects Referred to By Any Objects : \n")
print(heap_status1.referents)
print("\nSet of Objects Directly Refer to Any Objects : \n")
print(heap_status1.referrers)
print("\nStat Object : ")
print(heap_status1.stat)
print("First 3 Entries : ")
print("Index Count Size Cumulative Size Object Name")
for row in list(heap_status1.stat.get_rows())[:3]:
print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))
print("\nDumping Heap Status to a File : ")
heap_status1.dump("guppy_heap_status.out")
print(open("guppy_heap_status.out").readlines()[:5])
loaded_heap_stat = heap.load("guppy_heap_status.out")
print("First 3 Entries From Loaded File : ")
print("Index Count Size Cumulative Size Object Name")
for row in list(loaded_heap_stat.get_rows())[:3]:
print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))
OUTPUT
Few Important Properties/Methods of Heap Status Object(<class 'guppy.heapy.UniSet.IdentitySetMulti'>) :
Count of Objects in Heap : 98237
Heap Size : 12986124 bytes
Heap 1st Entry :
Partition of a set of 27446 objects. Total size = 4200962 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 27446 100 4200962 100 4200962 100 str
Set of Objects Referred to By Any Objects :
Partition of a set of 98230 objects. Total size = 12980396 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 27446 28 4200962 32 4200962 32 str
1 26392 27 2002320 15 6203282 48 tuple
2 13605 14 1021458 8 7224740 56 bytes
3 6771 7 978680 8 8203420 63 types.CodeType
4 6236 6 848096 7 9051516 70 function
5 940 1 801960 6 9853476 76 type
6 273 0 541800 4 10395276 80 dict of module
7 940 1 509240 4 10904516 84 dict of type
8 1198 1 476560 4 11381076 88 dict (no owner)
9 791 1 191880 1 11572956 89 dict of function
<248 more rows. Type e.g. '_.more' to view.>
Set of Objects Directly Refer to Any Objects :
Partition of a set of 52393 objects. Total size = 7348954 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 26396 50 2002832 27 2002832 27 tuple
1 6772 13 978824 13 2981656 41 types.CodeType
2 6236 12 848096 12 3829752 52 function
3 941 2 803016 11 4632768 63 type
4 273 1 541800 7 5174568 70 dict of module
5 940 2 509240 7 5683808 77 dict of type
6 810 2 388032 5 6071840 83 dict (no owner)
7 2113 4 169040 2 6240880 85 types.WrapperDescriptorType
8 454 1 111000 2 6351880 86 dict of function
9 49 0 98528 1 6450408 88 set
<219 more rows. Type e.g. '_.more' to view.>
Stat Object :
<guppy.heapy.Part.Stat object at 0x7fb7db895ba8>
First 3 Entries :
Index Count Size Cumulative Size Object Name
027446 4200962 4200962 str
126392 2002320 6203282 tuple
213605 1021458 7224740 bytes
Dumping Heap Status to a File :
['.loader: _load_stat\n', '.format: SetFormat\n', '.timemade: 1603777688.456192\n', '.count: 98230\n', '.size: 12985525\n']
First 3 Entries From Loaded File :
Index Count Size Cumulative Size Object Name
027439 4200493 4200493 str
126392 2002296 6202789 tuple
213605 1021352 7224141 bytes
Example 6¶
As a part of the sixth example, we have explained various attributes available through heap status object which lets us group heap status entries based on different attributes like, type, size, referrers, memory address, etc.
- bytype - This attribute of heap status groups heap status entries by type of object. All
dictentries will be combined into one entry. - byrcs - This attribute of heap status groups heap status entries by type of referrers.
- bymodule - This attribute of heap status groups heap status entries by the module.
- bysize - This attribute of heap status groups heap status entries by the individual size of object.
- byunity - This attribute of heap status groups heap status entries by total size.
- byvia -This attribute of heap status groups heap status entries by objects via which they are referred.
- byidset - This attribute of heap status groups heap status entries by idset.
- byid - This attribute of heap status groups heap status entries by memory address.
Below we have explained all attributes mentioned above one by one. We can easily see results to see how objects are grouped by.
CODE
from guppy import hpy
heap = hpy()
heap_status1 = heap.heap()
print("=========== Heap Status At Starting : ============")
print(heap_status1)
print("\n============ Heap Status Grouped By Type : ==========")
print(heap_status1.bytype)
print("\n==== Heap Status Grouped By Referrers of kind(class/dict of class) : ===")
print(heap_status1.byrcs)
print("\n========== Heap Status Grouped By Module : ============")
print(heap_status1.bymodule)
print("\n========== Heap Status Grouped By Individual Size : ==============")
print(heap_status1.bysize)
print("\n========= Heap Status Grouped By Total Size : ============")
print(heap_status1.byunity)
print("\n======= Heap Status Grouped By Referred Via : =============")
print(heap_status1.byvia)
print("\n========= Heap Status Grouped By IDset : ============")
print(heap_status1.byidset)
print("\n======== Heap Status Grouped By Address : ==========")
print(heap_status1.byid)
OUTPUT
=========== Heap Status At Starting : ============
Partition of a set of 38336 objects. Total size = 4307591 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 11218 29 988030 23 988030 23 str
1 9449 25 688544 16 1676574 39 tuple
2 2472 6 357512 8 2034086 47 types.CodeType
3 4922 13 343021 8 2377107 55 bytes
4 444 1 336552 8 2713659 63 type
5 2282 6 310352 7 3024011 70 function
6 444 1 244808 6 3268819 76 dict of type
7 101 0 174656 4 3443475 80 dict of module
8 253 1 117584 3 3561059 83 dict (no owner)
9 1090 3 87200 2 3648259 85 types.WrapperDescriptorType
<116 more rows. Type e.g. '_.more' to view.>
============ Heap Status Grouped By Type : ==========
Partition of a set of 38336 objects. Total size = 4307591 bytes.
Index Count % Size % Cumulative % Type
0 11218 29 988030 23 988030 23 str
1 9449 25 688544 16 1676574 39 tuple
2 1301 3 639080 15 2315654 54 dict
3 2472 6 357512 8 2673166 62 types.CodeType
4 4922 13 343021 8 3016187 70 bytes
5 444 1 336552 8 3352739 78 type
6 2282 6 310352 7 3663091 85 function
7 1090 3 87200 2 3750291 87 types.WrapperDescriptorType
8 41 0 73184 2 3823475 89 set
9 64 0 63344 1 3886819 90 abc.ABCMeta
<78 more rows. Type e.g. '_.more' to view.>
==== Heap Status Grouped By Referrers of kind(class/dict of class) : ===
Partition of a set of 38336 objects. Total size = 4307847 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 12186 32 891421 21 891421 21 types.CodeType
1 4454 12 491986 11 1383407 32 function
2 4396 11 480762 11 1864169 43 dict of type
3 1445 4 337424 8 2201593 51 type
4 4573 12 312848 7 2514441 58 tuple
5 1422 4 240269 6 2754710 64 dict of module
6 651 2 160561 4 2915271 68 function, tuple
7 1645 4 107779 3 3023050 70 set
8 53 0 101968 2 3125018 73 function, module
9 91 0 98608 2 3223626 75 dict of module, tuple, types.GetSetDescriptorType
<420 more rows. Type e.g. '_.more' to view.>
========== Heap Status Grouped By Module : ============
Partition of a set of 38336 objects. Total size = 4308103 bytes.
Index Count % Size % Cumulative % Module
0 38235 100 4300023 100 4300023 100 ~module
1 1 0 80 0 4300103 100 __main__
2 1 0 80 0 4300183 100 _abc
3 1 0 80 0 4300263 100 _bootlocale
4 1 0 80 0 4300343 100 _codecs
5 1 0 80 0 4300423 100 _collections
6 1 0 80 0 4300503 100 _ctypes
7 1 0 80 0 4300583 100 _functools
8 1 0 80 0 4300663 100 _heapq
9 1 0 80 0 4300743 100 _imp
<92 more rows. Type e.g. '_.more' to view.>
========== Heap Status Grouped By Individual Size : ==============
Partition of a set of 38336 objects. Total size = 4308103 bytes.
Index Count % Size % Cumulative % Individual Size
0 2427 6 349488 8 349488 8 144
1 2357 6 320552 7 670040 16 136
2 3735 10 268920 6 938960 22 72
3 248 1 261888 6 1200848 28 1056
4 4281 11 239736 6 1440584 33 56
5 2735 7 218800 5 1659384 39 80
6 2738 7 175232 4 1834616 43 64
7 108 0 127872 3 1962488 46 1184
8 415 1 99600 2 2062088 48 240
9 268 1 98624 2 2160712 50 368
<577 more rows. Type e.g. '_.more' to view.>
========= Heap Status Grouped By Total Size : ============
Partition of a set of 38336 objects. Total size = 4308103 bytes.
Index Count % Size % Cumulative % <unclassified>
0 38336 100 4308103 100 4308103 100 <Anything>
======= Heap Status Grouped By Referred Via : =============
Partition of a set of 38336 objects. Total size = 4308231 bytes.
Index Count % Size % Cumulative % Referred Via:
0 1021 3 423524 10 423524 10 '.__dict__'
1 2169 6 313800 7 737324 17 '.__code__'
2 2472 6 230886 5 968210 22 '.co_code'
3 2219 6 187992 4 1156202 27 '.co_names'
4 2436 6 179552 4 1335754 31 '.co_varnames'
5 2389 6 166768 4 1502522 35 '.co_consts'
6 628 2 157587 4 1660109 39 '.__doc__', '[0]'
7 402 1 142935 3 1803044 42 "['__doc__']"
8 1725 4 121048 3 1924092 45 '.__qualname__'
9 1661 4 108579 3 2032671 47 'list(_)[0]'
<6455 more rows. Type e.g. '_.more' to view.>
========= Heap Status Grouped By IDset : ============
Partition of a set of 38336 objects. Total size = 4308359 bytes.
Index Count % Size % Cumulative % by identity set
0 1 0 32992 1 32992 1 <1 set: 0x7fe8a7f38e48>
1 1 0 18528 0 51520 1 <1 dict of module: os>
2 1 0 9320 0 60840 1 <1 dict (no owner): 0x7fe8a93721f8*323>
3 1 0 9320 0 70160 2 <1 dict (no owner): 0x7fe8a941dca8*288>
4 1 0 9320 0 79480 2 <1 dict of module: posix>
5 1 0 8416 0 87896 2 <1 set: 0x7fe8a93e09e8>
6 1 0 8416 0 96312 2 <1 set: 0x7fe8a93e0c88>
7 1 0 7423 0 103735 2 <1 str: 'The class Bi... copy of S.\n'>
8 1 0 6036 0 109771 3 <1 str: "Support for ... 'error'.\n\n">
9 1 0 4791 0 114562 3 <1 str: 'Heap queues\...at Art! :-)\n'>
<38326 more rows. Type e.g. '_.more' to view.>
======== Heap Status Grouped By Address : ==========
Set of 38336 <mixed> objects. Total size = 4308359 bytes.
Index Size % Cumulative % Brief
0 32992 0.8 32992 0.8 set: 0x7fe8a7f38e48
1 18528 0.4 51520 1.2 dict of module: os
2 9320 0.2 60840 1.4 dict (no owner): 0x7fe8a93721f8*323
3 9320 0.2 70160 1.6 dict (no owner): 0x7fe8a941dca8*295
4 9320 0.2 79480 1.8 dict of module: posix
5 8416 0.2 87896 2.0 set: 0x7fe8a93e09e8
6 8416 0.2 96312 2.2 set: 0x7fe8a93e0c88
7 7423 0.2 103735 2.4 str: 'The class Bi... copy of S.\n'
8 6036 0.1 109771 2.5 str: "Support for ... 'error'.\n\n"
9 4791 0.1 114562 2.7 str: 'Heap queues\...at Art! :-)\n'
<38326 more rows. Type e.g. '_.more' to view.>
Example 7¶
As a part of this example, we'll explain two useful methods available through the heap object.
- iso() - It takes as input single object or multiple objects and returns status specifying object size. It considers complex objects like list, dict as one object.
- idset() - It takes as input single object or multiple objects and returns status specifying object size. It considers complex objects like list, dict as a list of the individual object.
Please feel free to go through the below example to see the difference between the results of the two methods.
CODE
import guppy
from guppy import hpy
import numpy as np
heap = hpy()
a = [i for i in range(1000000)]
b = "A"
c = "WORLD"
print("============== ISO Method Examples ====================")
print()
print(heap.iso(a))
print()
print(heap.iso(b))
print()
print(heap.iso(c))
print()
print(heap.iso([]))
print()
print(heap.iso(""))
print("\n============== IDSET Method Examples ===================")
print()
print(heap.idset(a))
print()
print(heap.idset(b))
print()
print(heap.idset(c))
print()
print(heap.idset([]))
print()
print(heap.idset(""))
OUTPUT
============== ISO Method Examples ====================
Partition of a set of 1 object. Total size = 8697464 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 8697464 100 8697464 100 list
Partition of a set of 1 object. Total size = 50 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 50 100 50 100 str
Partition of a set of 1 object. Total size = 54 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 54 100 54 100 str
Partition of a set of 1 object. Total size = 64 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 64 100 64 100 list
Partition of a set of 1 object. Total size = 49 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 49 100 49 100 str
============== IDSET Method Examples ===================
Partition of a set of 1000000 objects. Total size = 27999996 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1000000 100 27999996 100 27999996 100 int
Partition of a set of 1 object. Total size = 50 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 100 50 100 50 100 str
Partition of a set of 5 objects. Total size = 250 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 5 100 250 100 250 100 str
<Nothing>
<Nothing>
Example 8¶
As a part of our eighth and last example, we have demonstrated how we can check doc of particular method/attributes of objects available through guppy.
Below we have accessed the document of the heap by calling the doc attribute of a heap object. It'll show documentation for the whole heap object listing all methods/attributes available through that object. We have also accessed the document of individual method/attribute by calling that method attributes name on the doc attribute as explained below.
We have also accessed documentation of the heap status object. Please feel free to go through the below example to check how we can access docs of guppy objects.
CODE
from guppy import hpy
heap = hpy()
heap_status1 = heap.heap()
print("============== Heap Documents ====================")
print(heap.doc)
print("Doc On Individual Method/Function/Class : ")
print("\nISO Method Doc : \n")
print(heap.doc.iso)
print("\nRoot Class Doc : \n")
print(heap.doc.Root)
print("============= Heap Status Documents ================")
print(heap_status1.doc)
print("Doc On Individual Method/Function/Class : ")
print("\nDoc on parts Property of Heap Status : \n")
print(heap_status1.doc.parts)
print("\nDoc on byrcs Property of Heap Status : \n")
print(heap_status1.doc.byrcs)
OUTPUT
============== Heap Documents ====================
Top level interface to Heapy. Available attributes:
Anything Prod Via iso
Clodo Rcs doc load
Id Root findex monitor
Idset Size heap pb
Module Type heapu setref
Nothing Unity idset test
Use eg: heap.doc.<attribute> for info on <attribute>.
Doc On Individual Method/Function/Class :
ISO Method Doc :
iso( 0..*:node: Any+) -> IdentitySet[1]
Create a set of objects based on identity.
Argument
node: Any+
Any kind of objects are valid arguments.
Note
This method is the same as idset[2] except for the argument.
References
[0] heapy_Use.html#heapykinds.Use.iso
[1] heapy_UniSet.html#heapykinds.IdentitySet
[2] heapy_Use.html#heapykinds.Use.idset
Root Class Doc :
Root: RootStateType
This attribute is a symbolic root containing attributes from which all
reachable objects in the heap can be reached. It is the only value (a
singleton) of its kind; see [1] for a description of its attributes.
References
[0] heapy_Use.html#heapykinds.Use.Root
[1] heapy_RootState.html#heapykinds.RootStateType
============= Heap Status Documents ================
all byvia get_render pathsout
biper count get_rp prod
brief dictof get_shpaths referents
by diff imdom referrers
byclodo disjoint indisize rp
byid doc kind shpaths
byidset dominos maprox size
bymodule domisize more sp
byprod dump nodes stat
byrcs er owners test_contains
bysize fam partition theone
bytype get_ckc parts
byunity get_examples pathsin
Doc On Individual Method/Function/Class :
Doc on parts Property of Heap Status :
An iterable object, that can be used to iterate over the 'parts' of
self. The iteration order is determined by the sorting order the set
has, in the table printed when partitioned.
Doc on byrcs Property of Heap Status :
A copy of self, but with 'Rcs' as the equivalence relation.
This ends our simple tutorial explaining guppy usage. Please feel free to let us know your views in the comments section.
References¶
- How to Profile Memory Usage in Python using memory_profiler
- Garbage Collection in Python
- line_profiler: Line by Line Profiling of Python Code
- Snakeviz - Visualize Profiling Results in Python
- How to Profile Python Code using cProfile and Profile
- Scalene - CPU and Memory Profiler for Python Code
- pprofile - Deterministic & Statistical Profiler For Python Code
- tracemalloc - How to Trace Memory Usage in Python Code
- py-spy - Sampling Profiler for Python Code
- pyinstrument - Statistical Profiler for Python Code
- yappi - Yet Another Python Profiler
- pympler - Monitor Memory Usage by Python Objects
Sunny Solanki
Sunny Solanki
Intro: Software Developer | Youtuber | Bonsai Enthusiast
About: Sunny Solanki holds a bachelor's degree in Information Technology (2006-2010) from L.D. College of Engineering. Post completion of his graduation, he has 8.5+ years of experience (2011-2019) in the IT Industry (TCS). His IT experience involves working on Python & Java Projects with US/Canada banking clients. Since 2020, he’s primarily concentrating on growing CoderzColumn.
His main areas of interest are AI, Machine Learning, Data Visualization, and Concurrent Programming. He has good hands-on with Python and its ecosystem libraries.
Apart from his tech life, he prefers reading biographies and autobiographies. And yes, he spends his leisure time taking care of his plants and a few pre-Bonsai trees.
Contact: sunny.2309@yahoo.in
![YouTube Subscribe]()
Comfortable Learning through Video Tutorials?
If you are more comfortable learning through video tutorials then we would recommend that you subscribe to our YouTube channel.
![Need Help]()
Stuck Somewhere? Need Help with Coding? Have Doubts About the Topic/Code?
Stuck Somewhere? Need Help with Coding? Have Doubts About the Topic/Code?When going through coding examples, it's quite common to have doubts and errors.
If you have doubts about some code examples or are stuck somewhere when trying our code, send us an email at coderzcolumn07@gmail.com. We'll help you or point you in the direction where you can find a solution to your problem.
You can even send us a mail if you are trying something new and need guidance regarding coding. We'll try to respond as soon as possible.
![Share Views]()
Want to Share Your Views? Have Any Suggestions?
Want to Share Your Views? Have Any Suggestions?If you want to
- provide some suggestions on topic
- share your views
- include some details in tutorial
- suggest some new topics on which we should create tutorials/blogs
memoery-profiling, guppy, heapy
Subscribe to Our YouTube Channel
