Calculate Differences of IFC files with Hashing


 

 

Calculate Differences of IFC files with Hashing

Purpose

The goal of this tutorial is to learn how to asses whether 2 successively exported IFC building models are different by testing whether entity instances of the latest are present in the original one or entity instances of the original one are present in the latest version.

The data is stored using the Industry Foundation Class (IFC) format, which is the standard format to digitally represent anything related to the construction industry. This format stands as an open-standard, and enables to share building models in form of IFC files between the different actors of a construction project. We can think of IFC as the PDF for the construction industry.

For example, how could you find the difference between those 2 files?

image.png

Or even those ones, which are simpler.

image.png

But first, why would we want to compare two IFC files?

IFC files allow actors involved in a construction project to store an engineering model of the elements that form the building or the infrastructure modeled. In order to rely on this model, the data should be accurate or at least verifiable. Indeed, the models are not carved in stone and will know many changes throughout the building lifecycle, from the pre-design phase to the renovation or demolition phases.

It is therefore essential to provide software programs that let the users know whether they are receiving a newer version from their local copy, to be able to assess quantitatively and qualitatively the differences that have been introduced.

Given the complexity of a construction project in terms of the diversity of domains, the resulting number of actors, and some additional stakeholders, storing satisfying data for all in discipline related files is the daunting task IFC is trying to achieve.

Indeed, IFC files can become rather large and it is advised to use clever ways to process its data. The processing in question in this tutorial is to spot the differences among entity instances between two IFC files.

Before diving in, let’s review quickly some of the intricacies of the IFC format.

Background on the IFC format and comparison strategy

The IFC format defines a data schema divided into resources block. For each block, IFC entity types enable to model different elements. Those elements get an “Ifc” prefix and can be for example IfcWall, IfcSlab, IfcPropertySet, IfcRelDefinesByProperty, IfcRepresentation…

Whithin the data schema, which is object-oriented, there is a hierarchy between entity types, involving abstract entity types. For example, the IfcBuildingElement is an abstract entity type.

For example the defiinition of the IfcBuildingElement entity type is given below (source: http://standards.buildingsmart.org/MVD/RELEASE/IFC4/ADD2_TC1/RV1_2/HTML/schema/ifcproductextension/lexical/ifcbuildingelement.htm)

The building element comprises all elements that are primarily part of the construction of a building, i.e., its structural and space separating system. Building elements are all physically existent and tangible things

NOTE  Definition from ISO 6707-1:
Major functional part of a building, examples are foundation, floor, roof, wall.

image.png

The above diagram is an example of the inheritance links existing in the IFC data schema.

Moreover, an IFC file uses the entity types of the IFC schema with it own hierarchy.
It means the IFC instances implemented, like the 4 IfcWall instances of a very simple house, have relationships with other IFC instances. For example, the IfcWall instances are contained in an IfcBuildingStorey instance, itself contained in a IfcBuilding instance, itself contained in a Ifcsite instance, and itself eventually contained in an IfcProject instance. In addition, for example, the attributes of an IfcWall are often grouped in IfcPropertySet instances, which are for example related by an IfcRelDefinesByProperty instance. Relationships betweem instances are also objectified: they are also represented by IFC instances.

Download resources from the web

Let’s import the ifcopenshell python package and the IFC Duplex file.

Download ifcopenshell-python

In [1]:
import urllib.request
url = 'https://s3.amazonaws.com/ifcopenshell-builds/ifcopenshell-python-36-v0.6.0-0d93633-linux64.zip'
filename = 'ifcopenshell.zip'
urllib.request.urlretrieve(url, filename)
Out[1]:
('ifcopenshell.zip', <http.client.HTTPMessage at 0x1b4b63e6b48>)
In [2]:
import zipfile
with zipfile.ZipFile('ifcopenshell.zip', 'r') as zip_ref:
    zip_ref.extractall('ifcopenshell')

Download an IFC file

In [3]:
import urllib.request
url = 'http://openifcmodel.cs.auckland.ac.nz/_models/030811DuplexModel-IFC-2011-05-05.zip'
filename = 'Duplex_files.zip'
urllib.request.urlretrieve(url, filename)
Out[3]:
('Duplex_files.zip', <http.client.HTTPMessage at 0x1b4b6432f48>)
In [4]:
import zipfile
with zipfile.ZipFile('Duplex_files.zip', 'r') as zip_ref:
    zip_ref.extractall('Duplex')

Getting started

Once we’ve downloaded the resources, we can get started and import the main libraries we’ll be using.

In [6]:
import json
import math
import sys
import random
In [7]:
import ifcopenshell
import ifcopenshell.geom 

import OCC 
import OCC.BRepTools

We can open the IFC file we’ve dowloaded.

In [8]:
file = ifcopenshell.open('Duplex/Duplex_A_20110505.ifc')

And get the IfcBuildingElement entitity instances in a building_elements variable.

In [9]:
building_elements = file.by_type("IfcBuildingElement")

Let’s take 2 random entity instances to compare them.

In [10]:
random_entity1 = building_elements[4]
random_entity2 = building_elements[5]

Get the information of the instances. Generally speaking, this information as it is returned gives describes an instance, regarding to its physical placement, its geometry representation, and any other attached piece of information.

In [11]:
info1 = random_entity1.get_info(recursive=True, include_identifier=False, ignore={"GlobalId", "OwnerHistory"})
info2 = random_entity2.get_info(recursive=True, include_identifier=False, ignore={"GlobalId", "OwnerHistory"})

info3 = info1

Let’s pretty print the information of and an entity instance.

In [19]:
from IPython.core.display import HTML
import pprint
HTML("<pre style='white-space:pre;font-size:10px;line-height:14px'>{}</pre>".format(
    pprint.pformat(info1))
)
Out[19]:
{'Description': None,
 'Name': 'Basic Wall:Interior - Partition (92mm Stud):139783',
 'ObjectPlacement': {'PlacementRelTo': {'PlacementRelTo': {'PlacementRelTo': {'PlacementRelTo': None,
                                                                              'RelativePlacement': {'Axis': None,
                                                                                                    'Location': {'Coordinates': (0.0,
                                                                                                                                 0.0,
                                                                                                                                 0.0),
                                                                                                                 'type': 'IfcCartesianPoint'},
                                                                                                    'RefDirection': None,
                                                                                                    'type': 'IfcAxis2Placement3D'},
                                                                              'type': 'IfcLocalPlacement'},
                                                           'RelativePlacement': {'Axis': None,
                                                                                 'Location': {'Coordinates': (0.0,
                                                                                                              0.0,
                                                                                                              0.0),
                                                                                              'type': 'IfcCartesianPoint'},
                                                                                 'RefDirection': None,
                                                                                 'type': 'IfcAxis2Placement3D'},
                                                           'type': 'IfcLocalPlacement'},
                                        'RelativePlacement': {'Axis': None,
                                                              'Location': {'Coordinates': (0.0,
                                                                                           0.0,
                                                                                           0.0),
                                                                           'type': 'IfcCartesianPoint'},
                                                              'RefDirection': None,
                                                              'type': 'IfcAxis2Placement3D'},
                                        'type': 'IfcLocalPlacement'},
                     'RelativePlacement': {'Axis': None,
                                           'Location': {'Coordinates': (4.769999999999999,
                                                                        -10.30800000000003,
                                                                        0.0),
                                                        'type': 'IfcCartesianPoint'},
                                           'RefDirection': None,
                                           'type': 'IfcAxis2Placement3D'},
                     'type': 'IfcLocalPlacement'},
 'ObjectType': 'Basic Wall:Interior - Partition (92mm Stud):128360',
 'Representation': {'Description': None,
                    'Name': None,
                    'Representations': ({'ContextOfItems': {'ContextIdentifier': None,
                                                            'ContextType': 'Model',
                                                            'CoordinateSpaceDimension': 3,
                                                            'Precision': 1e-09,
                                                            'TrueNorth': None,
                                                            'WorldCoordinateSystem': {'Axis': None,
                                                                                      'Location': {'Coordinates': (0.0,
                                                                                                                   0.0,
                                                                                                                   0.0),
                                                                                                   'type': 'IfcCartesianPoint'},
                                                                                      'RefDirection': None,
                                                                                      'type': 'IfcAxis2Placement3D'},
                                                            'type': 'IfcGeometricRepresentationContext'},
                                         'Items': ({'Points': ({'Coordinates': (0.0,
                                                                                0.0),
                                                                'type': 'IfcCartesianPoint'},
                                                               {'Coordinates': (1.456000000000001,
                                                                                0.0),
                                                                'type': 'IfcCartesianPoint'}),
                                                    'type': 'IfcPolyline'},),
                                         'RepresentationIdentifier': 'Axis',
                                         'RepresentationType': 'Curve2D',
                                         'type': 'IfcShapeRepresentation'},
                                        {'ContextOfItems': {'ContextIdentifier': None,
                                                            'ContextType': 'Model',
                                                            'CoordinateSpaceDimension': 3,
                                                            'Precision': 1e-09,
                                                            'TrueNorth': None,
                                                            'WorldCoordinateSystem': {'Axis': None,
                                                                                      'Location': {'Coordinates': (0.0,
                                                                                                                   0.0,
                                                                                                                   0.0),
                                                                                                   'type': 'IfcCartesianPoint'},
                                                                                      'RefDirection': None,
                                                                                      'type': 'IfcAxis2Placement3D'},
                                                            'type': 'IfcGeometricRepresentationContext'},
                                         'Items': ({'Depth': 2.795000000000196,
                                                    'ExtrudedDirection': {'DirectionRatios': (0.0,
                                                                                              0.0,
                                                                                              1.0),
                                                                          'type': 'IfcDirection'},
                                                    'Position': {'Axis': None,
                                                                 'Location': {'Coordinates': (0.0,
                                                                                              0.0,
                                                                                              0.0),
                                                                              'type': 'IfcCartesianPoint'},
                                                                 'RefDirection': None,
                                                                 'type': 'IfcAxis2Placement3D'},
                                                    'SweptArea': {'Position': {'Location': {'Coordinates': (0.7280000000000004,
                                                                                                            -0.0),
                                                                                            'type': 'IfcCartesianPoint'},
                                                                               'RefDirection': {'DirectionRatios': (-1.0,
                                                                                                                    0.0),
                                                                                                'type': 'IfcDirection'},
                                                                               'type': 'IfcAxis2Placement2D'},
                                                                  'ProfileName': None,
                                                                  'ProfileType': 'AREA',
                                                                  'XDim': 1.456000000000001,
                                                                  'YDim': 0.1240000000000021,
                                                                  'type': 'IfcRectangleProfileDef'},
                                                    'type': 'IfcExtrudedAreaSolid'},),
                                         'RepresentationIdentifier': 'Body',
                                         'RepresentationType': 'SweptSolid',
                                         'type': 'IfcShapeRepresentation'}),
                    'type': 'IfcProductDefinitionShape'},
 'Tag': '139783',
 'type': 'IfcWallStandardCase'}

Intuitively, we can compare 2 entities using those info variables.

In [59]:
print(info1 == info2)
print(info1 == info3)
False
True

But what if we wanted to check whether an entity belongs to a set of entities. In other words, how do we test whether an entity of a new file we’ve received has been added or remained the same?

For this, we need to store the entities – still in the form of their information – to a set.

The cell below produces an error since a set must be made of immutable/hashable elements.

In [60]:
a_set1 = {info1, info2, info3}
a_set2 = {info1, info2}
print(a_set1 == a_set2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-60-9a4a8bf8df91> in <module>
----> 1 a_set1 = {info1, info2, info3}
      2 a_set2 = {info1, info2}
      3 print(a_set1 == a_set2)

TypeError: unhashable type: 'dict'

In order to reach our goal, which is to create a set of the information of entity instances, we will have to transform the data in which the information of an entity is stored, and which is mutable for now (a dictionary), into an immutable one (a tuple for example). For this, we will use a newly created class that instantiates and immutable data container, and for which we can tailor the tolerance for squashing floating values into a hash number.

Frozensets

Alternatively, we could also store the information on an entity as embedded frozenset, which is the immutable version of the python set.

In [61]:
info1 = random_entity1.get_info(include_identifier=False, recursive=True, return_type=frozenset)
info2 = random_entity2.get_info(include_identifier=False, recursive=True, return_type=frozenset)

We could create a mini set of those embedded frozensets to test membership of the entities.

In [62]:
new_set = {info1, info2}
print(info2 in new_set)
True

Let’s see some definitions about the mutable and hashable notions.

Definitions

Mutable

A data structure is mutable if it can be modified after its intitialization, such as a list, a set, a dictionary.
Conversely, a tuple, an integer, a string, or a frozenset cannot be modified after initialization, so they are immutable data types.

Hashable

A hashing function maps an immutable variable to an integer. This integer has to be unique and represents the input value in a more simpler form.
We say that a variable is hashable if it can be hashed with the hash method of its underliying data type. For this, the object must be immutable.
The input of the hashing function is unbounded and can be of any type as long as it is immutable. However, the output of the function is bounded (there is a finite number of hashes), which can lead to Hashing collision, happening when two different outputs share the same hash value. Consequently, the more the the input data, the more frequent an output hash risks to be and thus the higher the risk of Hashing collision.
In the example (https://en.wikipedia.org/wiki/Hash_table#/media/File:Hash_table_3_1_1_0_1_0_0_SP.svg) below, Hash collision happens for the hash value 02, which is produced by both John Smith and Sandra Dee.

image.png

Comparing 2 IFC files

In the context of comparing 2 files, if we look at the big picture we would say that we have to assess, for each entity instance of the new file, whether it belongs to the old file or not. To do so, the entities of the old file have to be stored in a container, such as a set.
Thus, the key operation of this tutorial is membership testing.

As said earlier, the information variable of each entity instance of the original file have to be stored in a set. But each of this informaton variable comes in the form of a dictionary. However, a set, as the keys of a dictionary, needs its elements to be immutable so that they can be hashable.

To do so, we create a new data container – a fuzzy dictionary – which will have, unlike the python dictionary, a hash method. To create this immutable/hashable data structure, we transform the input values coming in the form of a dictionary to a tuple, which is an immutable data structure which allows hashing.

Methods to create an immutable data structure

The first method enables to get the information of an IFC file. The second one transforms this information into an immutable data structure.

In [63]:
def info_as_dictionary(e):
    return e.get_info(recursive=True, include_identifier=False, ignore={"GlobalId", "OwnerHistory"})
    
def to_fuzzy(e):
    return fuzzy_dict(info_as_dictionary(e), guid=e.GlobalId)

Make the dictionaries immutable

Let’s create the fuzzy_dict data container.

In [64]:
class fuzzy_dict(object):
    def __init__(self, d, eps=1e-5, guid=None):
        self.d = d
        self.eps = eps
        self.guid = guid
        
        mod = - int(round(math.log10(eps * 100)))
        def c(v):
            if isinstance(v, float):
                # print "round(%r, %r) = %r" % (v, mod, round(v, mod))
                return round(v, mod)
            elif isinstance(v, dict):
                return tuple(sorted((k,c(v)) for k, v in v.items()))
            elif isinstance(v, (tuple, list)):
                return type(v)(map(c, v))
            else:
                return v
                
        self.h = hash(c(self.d))
        
    def __hash__(self):
        return self.h
         
    def __eq__(self, other):
    
        def eq(v1, v2):
            if type(v1) != type(v2):
                return False
            if isinstance(v1, float):
                if abs(v1 - v2) > self.eps: return False
            elif isinstance(v1, dict):
                if not (fuzzy_dict(v1, eps=self.eps) == fuzzy_dict(v2, eps=self.eps)): return False
            elif isinstance(v1, (tuple, list)):
                if len(v1) != len(v2): return False
                for a, b in zip(v1, v2):
                    if not eq(a, b): return False
            else:
                if v1 != v2: return False
            return True
        
        if set(self.d.keys()) != set(other.d.keys()): return
        
        for v1, v2 in ((self.d[k], other.d[k]) for k in self.d):
            if not eq(v1, v2): return False
            
        return True

Rounding the float values

As it can be seen, the eq method uses a mod variable to define tolerance in the comparison. This mod is calculated on the basis of an epsilon value.

Let

  • $latex int &bg=ffffff&s=2 $ be a floor or ceiling function
  • $latex round &bg=ffffff&s=2 $ be another floor or ceiling function
  • $latex log_{100} &bg=ffffff&s=2 $ be the logarithm base 10
  • $latex {\epsilon} &bg=ffffff&s=2 $ be the tolerance to compare two floating point numbers

The mod function can thus be expressed as

$latex mod(\epsilon)={\mbox{- int}}\left( round( log_{10}(100 \epsilon ) \right) &bg=ffffff&s=4 $

We need to round floats to a number of digital places so that we’re sure that f(v +- eps) == f(v) .

Create class to store the differences between 2 IFC files

In [65]:
info1_immut = to_fuzzy(random_entity1)
info2_immut = to_fuzzy(random_entity2)
info3_immut = info1_immut

a_set1_immut = {info1_immut, info2_immut, info3_immut}
a_set2_immut = {info1_immut}

print(a_set1_immut == {info1_immut, info2_immut})
print(a_set1_immut == a_set2_immut)
True
False

Create class to store the differences between 2 IFC files

The most important processing happens in the calculate_differences method. In this method, we will assign each entity instance to one or several of these groups: “unchanged”, “added”, “deleted”, via its GlobalId.

First, we can loop over the instances of the new file and check whether it belongs to the old file. The “belonging” or rather membership is tested with the hash of the information of the entity instance. If it does belong to the set of the hashes of the old files instances information, then it means it has been unchanged. It it doesn’t belong to the old file, it means it has been added.

Similarly, we can loop through the instances of the old file and checks whether an entity instance is still there. If it doesn’t, it means that it has been deleted or modified.

The particularity is that an entity instance that we’ll define by its GlobalId there (not its information) can be in both files. In that case, its GlobalId is saved in both the “deleted” and the “added” classes. It means that the information of the entity has been modified from the old file to the new one. So the GlobalId remains the same, but not the information, hence the membership in those 2 classes.

In [66]:
class Difference(object):
    def __init__(self,old,new):

        self.old_file = ifcopenshell.open(old)
        self.old_file_guids = set(i.GlobalId for i in self.old_file.by_type("IfcBuildingElement"))
        self.old_file_hashes = set()
        for guid in self.old_file_guids:
            entity = self.old_file[guid]
            info = info_as_dictionary(entity)
            immut_dict = to_fuzzy(entity)
            self.old_file_hashes.add(immut_dict)


        self.new_file = ifcopenshell.open(new)
        self.new_file_guids = set(i.GlobalId for i in self.new_file.by_type("IfcBuildingElement"))
        self.new_file_hashes = set()
        for guid in self.new_file_guids:
            entity = self.new_file[guid]
            info = info_as_dictionary(entity)
            immut_dict = to_fuzzy(entity)
            self.new_file_hashes.add(immut_dict)


        self.added_in_new = set()
        self.unchanged_in_new = set()
        self.deleted_from_old = set()

    def calculate_differences(self):
        for entity_guid in self.new_file_guids:
            entity = self.new_file[entity_guid]
            if to_fuzzy(entity) in self.old_file_hashes:
                #print('unchanged')
                self.unchanged_in_new.add(entity_guid)
            else:
                #print('added')
                self.added_in_new.add(entity_guid)

        for entity_guid in self.old_file_guids:
            entity = self.old_file[entity_guid]
            if to_fuzzy(entity) not in self.new_file_hashes:
                #print('deleted')
                self.deleted_from_old.add(entity_guid)

        return {'unchanged':self.unchanged_in_new,'added':self.added_in_new,'deleted':self.deleted_from_old}

Test the Difference class with 2 files

In the following code blocks, we’ll print and then display the result of the calculate_differences method. It enables to know if an entity instance as been added to the new file or if another one has been deleted from the old file.

In [67]:
import ifcopenshell
import ifcopenshell.geom 

filename1 = "./diff/261110ADT-Smiley-West-Project-14-10-2005.ifc"
filename2 = "./diff/261110ADT-Smiley-West-5-Buildings-14-10-2005.ifc"

difference = Difference(filename1,filename2)
differences = difference.calculate_differences()

print("Unchanged: ", len(differences['unchanged']))
print("Added: ", len(differences['added']))
print("Deleted: ", len(differences['deleted']))

present_in_both_files = set(differences['deleted'] & set(differences['added']))
Unchanged:  354
Added:  31
Deleted:  27

Using the python-occ viewer, we can display the entity instances of both files and color them according to the class they belong to. This code cannot be run in the notebook but feel free to test it in a regular python execution.

In [ ]:
import OCC 
import OCC.BRepTools

s = ifcopenshell.geom.settings()
s.set(s.USE_PYTHON_OPENCASCADE,True)
viewer = ifcopenshell.geom.utils.initialize_display()
viewer.set_bg_gradient_color(255, 255, 255, 255, 255, 255)



of = set(i for i in difference.old_file_guids)
nf = set(i for i in difference.new_file_guids)

of.update(nf)


for guid in of:
    entity = None

    if guid in differences['added']:
        clr = (0.239, 0.839, 0.372)
        entity = difference.new_file[guid]

    if guid in differences['unchanged']:
        clr = (0.105,0.196,1)
        entity = difference.new_file[guid]


    if guid in differences['deleted']:
        clr = (0.980, 0.419, 0.474)
        entity = difference.old_file[guid]

    if guid in differences['added'] and guid in differences['deleted']:
        clr = (1, 0.4, 0)
        entity = difference.new_file[guid]



    if entity:
        shape = ifcopenshell.geom.create_shape(s,entity)

        display = ifcopenshell.geom.utils.display_shape(shape,clr=clr)

        if guid in differences['unchanged']:
            ifcopenshell.geom.utils.set_shape_transparency(display,0.9)

    viewer.FitAll()



ifcopenshell.geom.utils.main_loop()

Let’s visualize the changes made on a minimalistic example.

image.png

Let’s add some walls and display the additions.

image.png

Let’s now delete the one this newly added wall, and also one of the windows.

image.png

Now, let’s make some changes regarding a wall position and a door position.

image.png

Finally, if we can see the real difference between the original file and the last version (after adding, deleting, and modifying)

image.png

Further work

For future work, we’ll see how to produce more detailed visualisations and go back to the history of the changes known by the biggest file of this tutorial.

WordPress conversion from Diff tutorial.ipynb by nb2wp v0.3.1

 

Leave a comment

Your email address will not be published. Required fields are marked *