More detailed walkthrough

See the simple walkthrough for a quicker introduction to the PyDDM package. Here, we will go into a little more depth.

Tip: Before working through this, go through the simple walkthrough.

Importing the necessary modules

We will ned to import numpy, matplotlib, and xarray. We will also import the PyDDM package.

[1]:
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt

import numpy as np #numerical python used for working with arrays, mathematical operations
import xarray as xr #package for labeling and adding metadata to multi-dimensional arrays

import sys
sys.path.append("../../PyDDM") #must point to the PyDDM folder
import ddm_analysis_and_fitting as ddm

Initializing DDM_Analysis class and computing the DDM matrix

The instance of the DDM_Analysis class we create will need, when initialized, metadata about the images to analyze and the analysis and fitting parameters. This can be done by using a yaml file as shown in the following cell of code (there, the metadata is saved in the file “example_data_silica_beads.yml”. But one can also initialize DDM_Analysis with a dictionary containing the necessary metadata. One way to create such a dictionary and then using it to initialize DDM_Analysis is shown below.

import yaml
ddm_analysis_parameters_str = """
DataDirectory: 'C:/Users/rmcgorty/Documents/GitHub/DDM-at-USD/ExampleData/'
FileName: 'images_nobin_40x_128x128_8bit.tif'
Metadata:
  pixel_size: 0.242 # size of pixel in um
  frame_rate: 41.7 #frames per second
Analysis_parameters:
  number_lag_times: 40
  last_lag_time: 600
  binning: no
Fitting_parameters:
  model: 'DDM Matrix - Single Exponential'
  Tau: [1.0, 0.001, 10]
  StretchingExp: [1.0, 0.5, 1.1]
  Amplitude: [1e2, 1, 1e6]
  Background: [2.5e4, 0, 1e7]
  Good_q_range: [5, 20]
  Auto_update_good_q_range: True
"""
parameters_as_dictionary = yaml.safe_load(ddm_analysis_parameters_str)
ddm_calc = ddm.DDM_Analysis(parameters_as_dictionary)
[2]:
#The yaml file `example_data_silica_beads.yml` contains the metadata and parameters above

ddm_calc = ddm.DDM_Analysis("../../Examples/example_data_silica_beads.yml")
Provided metadata: {'pixel_size': 0.242, 'frame_rate': 41.7}
Image shape: 3000-by-128-by-128
Number of frames to use for analysis: 3000
Maximum lag time (in frames): 600
Number of lag times to compute DDM matrix: 40

Below, with the method calculate_DDM_matrix, we compute the DDM matrix and some associated data. The data will be stored as an xarray Dataset as an attribute to ddm_calc called ddm_dataset.

Note: There are a few optional arguments we can pass to calculate_DDM_matrix. There is an optional argument quiet (True or False, False by default). Then we have some optional keyword arguments (all of which could also be specified in the YAML file). These are: overlap_method which sets the degree of overlap between image pairs when finding all image differences for a each lag time and is either 0, 1, 2, or 3, background_method which sets how to estimate the background parameter B and is either 0, 1, 2, or 3, and number_lag_times. If any of these three keyword arguments are set here, the value specified in the YAML file will be overwritten.

[3]:
ddm_calc.calculate_DDM_matrix()
2022-02-10 14:00:30,212 - DDM Calculations - Running dt = 1...
2022-02-10 14:00:37,571 - DDM Calculations - Running dt = 5...
2022-02-10 14:00:43,229 - DDM Calculations - Running dt = 9...
2022-02-10 14:00:47,878 - DDM Calculations - Running dt = 16...
2022-02-10 14:00:51,469 - DDM Calculations - Running dt = 27...
2022-02-10 14:00:54,913 - DDM Calculations - Running dt = 47...
2022-02-10 14:00:58,136 - DDM Calculations - Running dt = 81...
2022-02-10 14:01:01,219 - DDM Calculations - Running dt = 138...
2022-02-10 14:01:04,153 - DDM Calculations - Running dt = 236...
2022-02-10 14:01:07,126 - DDM Calculations - Running dt = 402...
DDM matrix took 39.838425159454346 seconds to compute.
 Background estimate ± std is 211.17 ± 1.49
[3]:
<xarray.Dataset>
Dimensions:           (lagtime: 40, q_y: 128, q_x: 128, q: 64, y: 128, x: 128, frames: 40)
Coordinates:
  * lagtime           (lagtime) float64 0.02398 0.04796 0.07194 ... 12.59 14.36
    framelag          (frames) int32 1 2 3 4 5 6 7 ... 308 352 402 459 525 599
  * q_y               (q_y) float64 -12.98 -12.78 -12.58 ... 12.37 12.58 12.78
  * q_x               (q_x) float64 -12.98 -12.78 -12.58 ... 12.37 12.58 12.78
  * q                 (q) float64 0.0 0.2028 0.4057 0.6085 ... 12.37 12.58 12.78
  * y                 (y) int32 0 1 2 3 4 5 6 7 ... 121 122 123 124 125 126 127
  * x                 (x) int32 0 1 2 3 4 5 6 7 ... 121 122 123 124 125 126 127
Dimensions without coordinates: frames
Data variables:
    ddm_matrix_full   (lagtime, q_y, q_x) float64 194.5 183.5 ... 192.0 196.8
    ddm_matrix        (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    first_image       (y, x) float64 128.0 149.0 173.0 ... 175.0 229.0 215.0
    avg_image_ft      (q) float64 0.0 1.293e+05 5.225e+03 ... 105.3 104.7 105.3
    num_pairs_per_dt  (lagtime) int32 2999 2998 2997 1498 1498 ... 20 17 15 13
    B                 float64 211.2
    B_std             float64 1.491
    Amplitude         (q) float64 -211.2 2.585e+05 1.024e+04 ... -1.699 -0.52
    ISF               (lagtime, q) float64 0.0 0.9997 0.9892 ... -4.952 -19.73
Attributes: (12/24)
    units:                   Intensity
    lagtime:                 sec
    q:                       μm$^{-1}$
    x:                       pixels
    y:                       pixels
    info:                    ddm_matrix is the averages of FFT difference ima...
    ...                      ...
    split_into_4_rois:       no
    use_windowing_function:  no
    binning:                 no
    bin_size:                2
    central_angle:           no
    angle_range:             no
_images/Walkthrough_-_details_8_3.png
_images/Walkthrough_-_details_8_4.png
_images/Walkthrough_-_details_8_5.png
_images/Walkthrough_-_details_8_6.png

Digging into the DDM xarray.Dataset

Let’s look at the dataset created after running calculate_DDM_matrix.

[4]:
display(ddm_calc.ddm_dataset)
<xarray.Dataset>
Dimensions:           (lagtime: 40, q_y: 128, q_x: 128, q: 64, y: 128, x: 128, frames: 40)
Coordinates:
  * lagtime           (lagtime) float64 0.02398 0.04796 0.07194 ... 12.59 14.36
    framelag          (frames) int32 1 2 3 4 5 6 7 ... 308 352 402 459 525 599
  * q_y               (q_y) float64 -12.98 -12.78 -12.58 ... 12.37 12.58 12.78
  * q_x               (q_x) float64 -12.98 -12.78 -12.58 ... 12.37 12.58 12.78
  * q                 (q) float64 0.0 0.2028 0.4057 0.6085 ... 12.37 12.58 12.78
  * y                 (y) int32 0 1 2 3 4 5 6 7 ... 121 122 123 124 125 126 127
  * x                 (x) int32 0 1 2 3 4 5 6 7 ... 121 122 123 124 125 126 127
Dimensions without coordinates: frames
Data variables:
    ddm_matrix_full   (lagtime, q_y, q_x) float64 194.5 183.5 ... 192.0 196.8
    ddm_matrix        (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    first_image       (y, x) float64 128.0 149.0 173.0 ... 175.0 229.0 215.0
    avg_image_ft      (q) float64 0.0 1.293e+05 5.225e+03 ... 105.3 104.7 105.3
    num_pairs_per_dt  (lagtime) int32 2999 2998 2997 1498 1498 ... 20 17 15 13
    B                 float64 211.2
    B_std             float64 1.491
    Amplitude         (q) float64 -211.2 2.585e+05 1.024e+04 ... -1.699 -0.52
    ISF               (lagtime, q) float64 0.0 0.9997 0.9892 ... -4.952 -19.73
Attributes: (12/24)
    units:                   Intensity
    lagtime:                 sec
    q:                       μm$^{-1}$
    x:                       pixels
    y:                       pixels
    info:                    ddm_matrix is the averages of FFT difference ima...
    ...                      ...
    split_into_4_rois:       no
    use_windowing_function:  no
    binning:                 no
    bin_size:                2
    central_angle:           no
    angle_range:             no
[5]:
print("Notice the data variables: \n", ddm_calc.ddm_dataset.data_vars)
Notice the data variables:
 Data variables:
    ddm_matrix_full   (lagtime, q_y, q_x) float64 194.5 183.5 ... 192.0 196.8
    ddm_matrix        (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    first_image       (y, x) float64 128.0 149.0 173.0 ... 175.0 229.0 215.0
    avg_image_ft      (q) float64 0.0 1.293e+05 5.225e+03 ... 105.3 104.7 105.3
    num_pairs_per_dt  (lagtime) int32 2999 2998 2997 1498 1498 ... 20 17 15 13
    B                 float64 211.2
    B_std             float64 1.491
    Amplitude         (q) float64 -211.2 2.585e+05 1.024e+04 ... -1.699 -0.52
    ISF               (lagtime, q) float64 0.0 0.9997 0.9892 ... -4.952 -19.73

Within this ddm_datset, we have the DDM matrix \(D(q_x, q_y, \Delta t)\) stored as ddm_matrix_full. We can take a look at a slice of this matrix with the matshow function in matplotlib.

[6]:
###############################################################
# Displaying a slice of the full three-dimensional DDM matrix #
###############################################################

lagtime_for_ddmmatrix = 10
fig,ax = plt.subplots()
plt.matshow(ddm_calc.ddm_dataset.ddm_matrix_full[lagtime_for_ddmmatrix], fignum=0, cmap='gray')

#Setting up the ticks and tick labels so that they match q_x and q_y
ticks = np.linspace(0,ddm_calc.ddm_dataset.ddm_matrix_full.shape[1]-1,7,dtype=int)
ticklabelsx = ["{:4.2f}".format(i) for i in ddm_calc.ddm_dataset.q_x[ticks].values]
ticklabelsy = ["{:4.2f}".format(i) for i in ddm_calc.ddm_dataset.q_y[ticks].values]
ax.set_xticks(ticks)
ax.set_xticklabels(ticklabelsx)
ax.set_yticks(ticks)
ax.set_yticklabels(ticklabelsy)

plt.title("DDM matrix for lag time of %.3f s" % ddm_calc.ddm_dataset.lagtime[10])
[6]:
Text(0.5, 1.0, 'DDM matrix for lag time of 0.288 s')
_images/Walkthrough_-_details_14_1.png

The slice of the DDM matrix above looks pretty radially symmetric. This is expected as we are analyzing a video of silica particles diffusing randomly with no preferred direction. Because of this symmetry, we can radially average the DDM matrix to go from \(D(q_x, q_y, \Delta t)\) to \(D(q, \Delta t)\) where \(q = \sqrt{q_x^2 + q_y^2}\).

Let’s plot a slice of this radially averaged DDM matrix \(D(q, \Delta t)\) (which is stored as the variable ddm_matrix in the ddm_dataset).

[7]:
######################################################################
# Plotting the radially averaged DDM matrix for a particular q-value #
######################################################################

q_index = 15 #index of the list of q values
plt.figure(figsize=(10,10/1.618))
plt.semilogx(ddm_calc.ddm_dataset.lagtime, ddm_calc.ddm_dataset.ddm_matrix[:,q_index], 'ro')
plt.xlabel("Lag time (s)")
plt.ylabel("DDM matrix")
plt.title("DDM matrix for $q$ = %.3f $\mu$m$^{-1}$" % ddm_calc.ddm_dataset.q[q_index]);
_images/Walkthrough_-_details_16_0.png

Different data selecting methods

Since we are slicing this xarray dataset, let’s use the opportunity to go over different ways of selecting data stored in a dataset. See the xarray user guide for more info.

  1. We can use the isel function (see xarray.DataArray.isel).

q_index = 15 #index of the list of q values
plt.figure(figsize=(10,10/1.618))
plt.semilogx(ddm_calc.ddm_dataset.lagtime, ddm_calc.ddm_dataset.ddm_matrix.isel(q=q_index), 'ro')
plt.xlabel("Lag time (s)")
plt.ylabel("DDM matrix")
plt.title("DDM matrix for q = %.3f " % ddm_calc.ddm_dataset.q[q_index])
  1. We can select the desired q not based on its index but on its actual value with sel (see xarray.DataArray.sel. Note that here we have to use method = 'nearest' since we do not have a q value of exactly 3 \(\mu\)m\(^{-1}\).

q_index = 15 #index of the list of q values
plt.figure(figsize=(10,10/1.618))
plt.semilogx(ddm_calc.ddm_dataset.lagtime, ddm_calc.ddm_dataset.ddm_matrix.sel(q=3, method='nearest'), 'ro')
plt.xlabel("Lag time (s)")
plt.ylabel("DDM matrix")
plt.title("DDM matrix for q = %.3f " % ddm_calc.ddm_dataset.q[q_index])
  1. Lastly, we can create this plot (using whichever slicing method) with xarray’s plotting function (which just wraps matplotlib.pyplot). For more, see xarray.DataArray.plot.

q_index = 15
ddm_calc.ddm_dataset.ddm_matrix.isel(q=q_index).plot(xscale='log', marker='o', ls='none')

Determing the A and B parameters to get the ISF

Let’s now look at the avg_image_ft data variable. This is a radially averaged average of all the Fourier transforms of the images (i.e., the radially average of \(\left< | \tilde{I}(q, t) |^2 \right>_t\)). This function should equal $ (1/2) \times `:nbsphinx-math:left[ A(q) + B(q) right] `$.

If we make the assumption that at the highest wavevectors (large \(q\)) the amplitude, \(A(q)\), goes to zero, then we can estimate the background, \(B\). If we also assume that the background is a constant that’s independent of \(q\), then we can determine \(A(q)\).

Plotted below is this avg_image_ft along with a horizontal line which can be used to estimate half the background. This estimate is taken from the last 10% of the wavevectors. The background can be saved to the B data variable. And then we can determine the amplitude which is saved as the Amplitude variable.

Note: What is described above is the default method for getting the background. Other methods can be selected with the optional parameter ‘background_method’ passed to the method ‘calculate_DDM_matrix’.

[8]:
############################################################################################
# Plotting the average squared Fourier-transformed image as a function of the wavevector q #
############################################################################################

plt.figure(figsize=(10,10/1.618))
plt.semilogy(ddm_calc.ddm_dataset.q[1:], ddm_calc.ddm_dataset.avg_image_ft[1:], 'ro')
plt.xlabel("Wavevector, q ($\mu$m$^{-1}$)")
plt.ylabel(r"$\left< | \tilde{I}(q, t) |^2 \right>_t$", fontsize=14)
plt.title("Default method to estimate the background and find the amplitude")

number_of_hi_qs = int(0.1*len(ddm_calc.ddm_dataset.q)) #for getting highest 10% of q's
plt.axhline(y = ddm_calc.ddm_dataset.avg_image_ft[-1*number_of_hi_qs:].mean())
[8]:
<matplotlib.lines.Line2D at 0x2299c86dbb0>
_images/Walkthrough_-_details_21_1.png

The other methods for getting the background (and from the background then estimating the amplitude) are: * background_method = 1: \(B\) will be the minimum of the DDM matrix * background_method = 2: \(B\) will be the average (over all lag times) of the DDM matrix at the largest wavevector \(q\) * background_method = 3: \(B\) = 0

If we know \(A(q)\) and \(B\), then we can find the intermediate scattering function, \(f(q, \Delta t)\), by assuming that the DDM matrix is equal to \(D (q, \Delta t) = A(q) \times \left[ 1 - f(q, \Delta t) \right] + B\).

The ISF is stored as the data variable ISF in the ddm_dataset.

[9]:
#############################################################################################################
# Plotting the intermediate scattering function (ISF) for a particular value of q as a function of lag time #
#############################################################################################################

plt.figure(figsize=(10,10/1.618))
plt.semilogx(ddm_calc.ddm_dataset.lagtime, ddm_calc.ddm_dataset.ISF.sel(q=1.5, method='nearest'), 'ro')
plt.xlabel("Lag time (s)")
plt.ylabel("ISF")
plt.title("ISF for q = %.3f $\mu$m$^{-1}$" % ddm_calc.ddm_dataset.q.sel(q=1.5, method='nearest'));
_images/Walkthrough_-_details_24_0.png

Checking the background estimate

We expect the ISF to go from 1 to 0. We may not see exactly this behavior in the ISF for all wavevectors. At larger wavevectors, the dynamics might be fast and we might not be taking images fast enough to see the early decay of the ISF. Conversely, at lower wavevectors, the dynamics might be slow and we may not be able to calculate the ISF for large enough lag times to see it decay all the way to zero. Additionally, for non-ergodic dynamics, the ISF may not decay to zero but to some value sometimes referred to as the non-ergodicity parameter.

But another reason that the ISF may not go from 1 to 0 is that we did a poor job in estimating the background, \(B\). Using the default method (background_method = 0), we try to estimate \(B\) by looking at the average Fourier transforms of all the images, radially averaging that, and assuming that the amplitude goes to zero at large wavevectors. But another way to estimate \(B\) is to look at the DDM matrix at large wavevectors, as is plotted below.

[10]:
###################################################################################################
# Plotting the radially averaged DDM matrix for the greatest value of q as a function of lag time #
###################################################################################################

q_index = -1 #index of the list of q values
plt.figure(figsize=(10,10/1.618))
plt.semilogx(ddm_calc.ddm_dataset.lagtime, ddm_calc.ddm_dataset.ddm_matrix[:,q_index], 'ro')
plt.xlabel("Lag time (s)")
plt.ylabel("DDM matrix")
plt.title("DDM matrix for $q$ = %.3f $\mu$m$^{-1}$" % ddm_calc.ddm_dataset.q[q_index]);
plt.axhline(y = ddm_calc.ddm_dataset.B)
[10]:
<matplotlib.lines.Line2D at 0x229a6721f70>
_images/Walkthrough_-_details_27_1.png

The DDM matrix should go from the background, \(B\), to the background plus amplitude, \(B + A\), as a function of the lag time. From the above plot, it does look like we overestimated the background (perhaps the amplitude hadn’t gone quite to zero by the last wavevectors we measure?). From the plot, clearly the background is less than 195.

Let’s estimate the background at about 190. We can make this adjustment with the recalculate_ISF_with_new_background function.

[11]:
new_ddm_matrix = ddm.recalculate_ISF_with_new_background(ddm_calc.ddm_dataset, background_val=190)
display(new_ddm_matrix)
<xarray.Dataset>
Dimensions:           (lagtime: 40, q_y: 128, q_x: 128, q: 64, y: 128, x: 128, frames: 40)
Coordinates:
  * lagtime           (lagtime) float64 0.02398 0.04796 0.07194 ... 12.59 14.36
    framelag          (frames) int32 1 2 3 4 5 6 7 ... 308 352 402 459 525 599
  * q_y               (q_y) float64 -12.98 -12.78 -12.58 ... 12.37 12.58 12.78
  * q_x               (q_x) float64 -12.98 -12.78 -12.58 ... 12.37 12.58 12.78
  * q                 (q) float64 0.0 0.2028 0.4057 0.6085 ... 12.37 12.58 12.78
  * y                 (y) int32 0 1 2 3 4 5 6 7 ... 121 122 123 124 125 126 127
  * x                 (x) int32 0 1 2 3 4 5 6 7 ... 121 122 123 124 125 126 127
Dimensions without coordinates: frames
Data variables:
    ddm_matrix_full   (lagtime, q_y, q_x) float64 194.5 183.5 ... 192.0 196.8
    ddm_matrix        (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    first_image       (y, x) float64 128.0 149.0 173.0 ... 175.0 229.0 215.0
    avg_image_ft      (q) float64 0.0 1.293e+05 5.225e+03 ... 105.3 104.7 105.3
    num_pairs_per_dt  (lagtime) int32 2999 2998 2997 1498 1498 ... 20 17 15 13
    B                 int32 190
    B_std             float64 1.491
    Amplitude         (q) float64 -190.0 2.585e+05 1.026e+04 ... 19.47 20.65
    ISF               (lagtime, q) float64 0.0 0.9996 0.9872 ... 0.432 0.4966
Attributes: (12/24)
    units:                   Intensity
    lagtime:                 sec
    q:                       μm$^{-1}$
    x:                       pixels
    y:                       pixels
    info:                    ddm_matrix is the averages of FFT difference ima...
    ...                      ...
    split_into_4_rois:       no
    use_windowing_function:  no
    binning:                 no
    bin_size:                2
    central_angle:           no
    angle_range:             no

Now let’s save this ddm_dataset. When the calculate_DDM_matrix method is executed, the ddm_dataset is automatically saved to disk. But we can overwrite that with the resave_ddm_dataset method.

[12]:
ddm_calc.resave_ddm_dataset(new_ddm_matrix)

Additional notes about the background

The default way that this code estimates the background is to compute the power spectrum of the images, as described above. That is, we find \(\left< | \tilde{I}(q, t) |^2 \right>_t\) and recognize that \(\left< | \tilde{I}(q, t) |^2 \right>_t = 0.5 \times \left[ A(q) + B \right]\) and assume that as \(q \rightarrow q_{max}\), \(A(q) \rightarrow 0\). Therefore, \(\left< | \tilde{I}(q_{max}, t) |^2 \right>_t \approx 0.5\times B\) (see [Cerbino 2017] or Eq. 5 in [Giavazzi 2018]).

But, there are other ways. In [Kurzthaler 2018], they set the background, \(B\), to zero. They note that for their fluorescence imaging “the camera noise is usually negligible.” However, they note that for bright-field of phase contrast modalities, the camera is not negligible and \(B\) will be non-zero.

In [Bayles 2017], they approximate \(B\) by \(\min \left[ D(q, \Delta t_{min}) \right]\). But they note that the frame rate needs to be “sufficiently small” and that this will give an overestimate of \(B\). To try this, use background_method = 1 as an optional keyword parameter in the function calculate_DDM_matrix (or set the value of background_method in the YAML file under Analysis_parameters).

[Cerbino 2017] Cerbino, R., Piotti, D., Buscaglia, M. & Giavazzi, F. Dark field differential dynamic microscopy enables accurate characterization of the roto-translational dynamics of bacteria and colloidal clusters. J. Phys.: Condens. Matter 30, 025901 (2017).

[Giavazzi 2018] Giavazzi, F., Malinverno, C., Scita, G. & Cerbino, R. Tracking-Free Determination of Single-Cell Displacements and Division Rates in Confluent Monolayers. Front. Phys. 6, (2018).

[Kurzthaler 2018] Kurzthaler, C. et al. Probing the Spatiotemporal Dynamics of Catalytic Janus Particles with Single-Particle Tracking and Differential Dynamic Microscopy. Phys. Rev. Lett. 121, 078001 (2018).

[Bayles 2017] Bayles, A. V., Squires, T. M. & Helgeson, M. E. Probe microrheology without particle tracking by differential dynamic microscopy. Rheol Acta 56, 863–869 (2017).

Initiazing DDM_Fit class and fitting our data to a model

We initlize the DDM_Fit class by passing it the yaml file containing our analysis parameters.

[13]:
#Note that to initialize the DDM_Fit class we need to pass it the YAML file containing the parameters
# *or* a dictionary with the same parameters.

ddm_fit = ddm.DDM_Fit(ddm_calc.data_yaml)
Initial guess Minimum Maximum
Amplitude 100.0 1.000 1000000.0
Tau 1.0 0.001 10.0
Background 25000.0 0.000 10000000.0
StretchingExp 1.0 0.500 1.1
Loading file C:/Users/rmcgorty/Documents/GitHub/PyDDM/Examples/images_nobin_40x_128x128_8bit_ddmmatrix.nc ...

Based on what was done above in terms of estimating the background, we found that \(B\) should be about 190. We use that estimate of the background to determine \(A(q)\) and the intermediate scattering function, \(f(q,\Delta t)\). We could proceed to then fit the ISF. But here we choose not to. We fit the DDM matrix and have the amplitude and background, \(A\) and \(B\), be fitting parameters. But we can constrain these parameters based on what we found above.

[14]:
#We update the intial guess for the background and the lower and upper bounds
ddm_fit.set_parameter_initial_guess('Background', 190)
ddm_fit.set_parameter_bounds('Background', [10, 220])
Parameter 'Background' set to 190.
Parameter 'Background' lower limit set to 10.
Parameter 'Background' upper limit set to 220.

Our first attemp at fitting the data

When we execute the fit below, we can use the optional argument use_A_from_images_as_guess set to True. This will use the value of \(A(q)\) we esimated from the average Fourier tranform of all images and the estimated background. We can further limit the bounds on \(A(q)\) with the optional parameter update_limits_on_A set to True. This will restrict the range that \(A(q)\) can take to be within 10% of the estimated \(A(q)\). If we want the bounds to be stricter or looser than 10%, we can change the optional parameter updated_lims_on_A_fraction to be something other than 0.1 (the default).

[15]:
#Note that the method `fit` has many optional parameters

fit01 = ddm_fit.fit(name_fit = 'fit01', display_table=False, use_A_from_images_as_guess=True, update_limits_on_A=True)
In function 'get_tau_vs_q_fit', using new tau...
Fit is saved in fittings dictionary with key 'fit01'.
[16]:
#Generate plots for inspecting the outcome of the fits

ddm.fit_report(fit01, q_indices=[3,6,9,22], forced_qs=[4,16], use_new_tau=True, show=True)
In function 'get_tau_vs_q_fit', using new tau...
In hf.plot_one_tau_vs_q function, using new tau...
[16]:
<xarray.Dataset>
Dimensions:          (parameter: 4, q: 64, lagtime: 40)
Coordinates:
  * parameter        (parameter) <U13 'Amplitude' 'Tau' ... 'StretchingExp'
  * q                (q) float64 0.0 0.2028 0.4057 0.6085 ... 12.37 12.58 12.78
  * lagtime          (lagtime) float64 0.02398 0.04796 0.07194 ... 12.59 14.36
Data variables:
    parameters       (parameter, q) float64 1.0 2.326e+05 ... 0.705 0.7446
    theory           (lagtime, q) float64 10.0 315.0 338.5 ... 206.7 206.8 206.9
    isf_data         (lagtime, q) float64 0.0 0.9996 0.9872 ... 0.432 0.4966
    ddm_matrix_data  (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    A                (q) float64 -190.0 2.585e+05 1.026e+04 ... 19.47 20.65
    B                int32 190
Attributes: (12/18)
    model:                          DDM Matrix - Single Exponential
    data_to_use:                    DDM Matrix
    initial_params_dict:            ["{'n': 0, 'value': 100.0, 'limits': [1.0...
    effective_diffusion_coeff:      0.6168642315520588
    tau_vs_q_slope:                 [-1.98005917]
    msd_alpha:                      [1.01007082]
    ...                             ...
    DataDirectory:                  C:/Users/rmcgorty/Documents/GitHub/PyDDM/...
    FileName:                       images_nobin_40x_128x128_8bit.tif
    pixel_size:                     0.242
    frame_rate:                     41.7
    BackgroundMethod:               None
    OverlapMethod:                  2
_images/Walkthrough_-_details_42_2.png
_images/Walkthrough_-_details_42_3.png
_images/Walkthrough_-_details_42_4.png
_images/Walkthrough_-_details_42_5.png
_images/Walkthrough_-_details_42_6.png
_images/Walkthrough_-_details_42_7.png

Our second attemp at fitting the data - updating initial guesses for tau

From the results of the fit, we can estimate a diffusion coefficient of about 0.6 \(\mu m^2 / s\). We determine this by looking at the characteristic decay time vs the wavevector (\(\tau\) vs \(q\)). On a log-log plot, power laws relationships between the two variables will come across as a straight line where the slope of the line is equal to the power. And, for this data, we see a clear power law show a relationship of \(\tau \propto q^{-2}\) which indicates diffusive motion. Note that our data deviates from this power law relationship for \(q\) greater than about 4 \(\mu m^{-1}\). This could be due to poor optical resolution limiting the high wavevectors (which correspond to small length scales). Or it could be due to not having the temporal resolution necessary to detect fast decay times. Lastly, it could also be due to poor fit performance. We could try to use an initial guess for \(\tau\) for each \(q\) that is based on an estimated diffusion coefficient. That is, when we fit \(D(q, \Delta t)\) (or \(f(q,\Delta t)\)), for each \(q\), we will calculate \(\tau_{guess} = 1 / (Dq^2)\) and use that as the initial guess in the fits. We do that below.

[17]:
fit02 = ddm_fit.fit(name_fit = 'fit02', display_table=False, use_A_from_images_as_guess=True, update_limits_on_A=True,
                   update_tau_based_on_estimated_diffcoeff=True, estimated_diffcoeff=0.6,
                   update_limits_on_tau=True, updated_lims_on_tau_fraction=1)
In function 'get_tau_vs_q_fit', using new tau...
Fit is saved in fittings dictionary with key 'fit02'.
[18]:
ddm.fit_report(fit02, q_indices=[3,6,9,22], forced_qs=[4,16], use_new_tau=False, show=True)
[18]:
<xarray.Dataset>
Dimensions:          (parameter: 4, q: 64, lagtime: 40)
Coordinates:
  * parameter        (parameter) <U13 'Amplitude' 'Tau' ... 'StretchingExp'
  * q                (q) float64 0.0 0.2028 0.4057 0.6085 ... 12.37 12.58 12.78
  * lagtime          (lagtime) float64 0.02398 0.04796 0.07194 ... 12.59 14.36
Data variables:
    parameters       (parameter, q) float64 1.0 2.326e+05 9.894e+03 ... 0.5 0.5
    theory           (lagtime, q) float64 10.0 152.4 361.6 ... 202.0 201.7 201.6
    isf_data         (lagtime, q) float64 0.0 0.9996 0.9872 ... 0.432 0.4966
    ddm_matrix_data  (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    A                (q) float64 -190.0 2.585e+05 1.026e+04 ... 19.47 20.65
    B                int32 190
Attributes: (12/18)
    model:                          DDM Matrix - Single Exponential
    data_to_use:                    DDM Matrix
    initial_params_dict:            ["{'n': 0, 'value': 100.0, 'limits': [1.0...
    effective_diffusion_coeff:      0.6584064996820138
    tau_vs_q_slope:                 [-1.89886818]
    msd_alpha:                      [1.053259]
    ...                             ...
    DataDirectory:                  C:/Users/rmcgorty/Documents/GitHub/PyDDM/...
    FileName:                       images_nobin_40x_128x128_8bit.tif
    pixel_size:                     0.242
    frame_rate:                     41.7
    BackgroundMethod:               None
    OverlapMethod:                  2
_images/Walkthrough_-_details_46_1.png
_images/Walkthrough_-_details_46_2.png
_images/Walkthrough_-_details_46_3.png
_images/Walkthrough_-_details_46_4.png
_images/Walkthrough_-_details_46_5.png
_images/Walkthrough_-_details_46_6.png

It doesn’t appear that that helped much. In fact, we see that the \(\tau\) values appear to be “hitting the rails” of the bounds we imposed on the parameter.

Below, we plot \(\tau\) vs \(q\) for the last two fittings. We see that updating the initial guess didn’t improve things. So the reason that the \(\tau\) found for the high wavevectors is off does not seem to be related to the fitting algorithm using a poor initial guess. Perhaps at the high wavevectors the dynamics are too fast for us detect with the camera frame rate we used? Below we also plot a horizontal line corresponding the time interval between frames. We see that the fits are off even well above our temporal resolution. So it is also not the case that the fits at some of the high wavevectors is due to poor temporal resolution.

[19]:
################################################################################################
# Plotting the characteristic decay time tau vs wavevector q determined from the last two fits #
################################################################################################

plt.figure(figsize=(10,10/1.618))
plt.loglog(fit01.q, fit01.parameters.loc['Tau'], 'ro', label="Not updating tau's initial guess")
plt.loglog(fit02.q, fit02.parameters.loc['Tau'], 'bo', label="Updating tau's initial guess based on estimated diffusion coeff")
plt.axhline(y = 1.0/fit01.frame_rate)
plt.legend(loc=0)
[19]:
<matplotlib.legend.Legend at 0x229a892e760>
_images/Walkthrough_-_details_49_1.png

Our third attemp at fitting the data - tightening the bounds on the stretching exponent

Let’s try one more thing with this fitting. Let’s think about the stretching exponent. For normal diffusive motion, we expect the intermediate scattering function to be a simple exponential: \(f(q,\Delta t) = \exp(-\Delta t / \tau)\). But sometimes when the dynamics are heterogeneous or subdiffusion is present, we see that a stretched exponential works better. That is, for the ISF we use: \(f(q,\Delta t) = \exp(-\Delta t / \tau)^s\) where \(s\) is the stretching exponent.

For this example data, we just have movie of silica colloidal particles diffusing in water. The dynamics should be diffusive. So we could try constraining the stretching exponent to be around to 1 as we expect (notice how the stretching exponent is between 0.6 and 0.7 for the high \(q\) values in the previous fits).

[20]:
#We update the intial guess for the background and the lower and upper bounds
ddm_fit.set_parameter_initial_guess('StretchingExp', 1)
ddm_fit.set_parameter_bounds('StretchingExp', [0.85, 1])
Parameter 'StretchingExp' set to 1.
Parameter 'StretchingExp' lower limit set to 0.85.
Parameter 'StretchingExp' upper limit set to 1.
[21]:
fit03 = ddm_fit.fit(name_fit = 'fit03', display_table=False, use_A_from_images_as_guess=True, update_limits_on_A=True,
                   update_tau_based_on_estimated_diffcoeff=True, estimated_diffcoeff=0.6,
                   update_limits_on_tau=True, updated_lims_on_tau_fraction=1)
In function 'get_tau_vs_q_fit', using new tau...
Fit is saved in fittings dictionary with key 'fit03'.
[22]:
ddm.fit_report(fit03, q_indices=[3,6,9,22], forced_qs=[4,16], use_new_tau=False, show=True)
[22]:
<xarray.Dataset>
Dimensions:          (parameter: 4, q: 64, lagtime: 40)
Coordinates:
  * parameter        (parameter) <U13 'Amplitude' 'Tau' ... 'StretchingExp'
  * q                (q) float64 0.0 0.2028 0.4057 0.6085 ... 12.37 12.58 12.78
  * lagtime          (lagtime) float64 0.02398 0.04796 0.07194 ... 12.59 14.36
Data variables:
    parameters       (parameter, q) float64 1.0 2.326e+05 ... 0.85 0.85
    theory           (lagtime, q) float64 10.0 78.85 304.9 ... 200.8 200.6 200.4
    isf_data         (lagtime, q) float64 0.0 0.9996 0.9872 ... 0.432 0.4966
    ddm_matrix_data  (lagtime, q) float64 0.0 294.2 321.4 ... 207.8 201.1 200.4
    A                (q) float64 -190.0 2.585e+05 1.026e+04 ... 19.47 20.65
    B                int32 190
Attributes: (12/18)
    model:                          DDM Matrix - Single Exponential
    data_to_use:                    DDM Matrix
    initial_params_dict:            ["{'n': 0, 'value': 100.0, 'limits': [1.0...
    effective_diffusion_coeff:      0.6779463827208101
    tau_vs_q_slope:                 [-1.84532034]
    msd_alpha:                      [1.08382266]
    ...                             ...
    DataDirectory:                  C:/Users/rmcgorty/Documents/GitHub/PyDDM/...
    FileName:                       images_nobin_40x_128x128_8bit.tif
    pixel_size:                     0.242
    frame_rate:                     41.7
    BackgroundMethod:               None
    OverlapMethod:                  2
_images/Walkthrough_-_details_54_1.png
_images/Walkthrough_-_details_54_2.png
_images/Walkthrough_-_details_54_3.png
_images/Walkthrough_-_details_54_4.png
_images/Walkthrough_-_details_54_5.png
_images/Walkthrough_-_details_54_6.png
[23]:
##################################################################################################
# Plotting the characteristic decay time tau vs wavevector q determined from the last three fits #
##################################################################################################

plt.figure(figsize=(10,10/1.618))
plt.loglog(fit01.q, fit01.parameters.loc['Tau'], 'ro', label="Not updating tau's initial guess")
plt.loglog(fit02.q, fit02.parameters.loc['Tau'], 'bo', label="Updating tau's initial guess")
plt.loglog(fit03.q, fit03.parameters.loc['Tau'], 'go', label="Restricting the stretching exponent")
plt.xlabel("$q$ ($\mu$m$^{-1})$")
plt.ylabel("Decay time, $\\tau$ (s)")
plt.axhline(y = 1.0/fit01.frame_rate)
plt.legend(loc=0)
[23]:
<matplotlib.legend.Legend at 0x229a6934640>
_images/Walkthrough_-_details_55_1.png

What to make of the results at high \(q\)

So we have tried a couple things to improve the results of our fits for \(q \gtrsim 6 \mu m^{-1}\). We tried: 1. Updating the intial guess for \(\tau\) for each \(q\) based on an expected diffusion coefficient and constraining the bounds of \(\tau\) around that guess. 2. Narrowing the bounds of the stretching exponent to be close to what we would expect for normal diffusive motion (i.e., near 1).

But, these didn’t seem to make the result for \(\tau (q)\) reasonable for the larger \(q\) values. So what is going on?

The horizontal line in the plot above corresponds to the minimum time lag we are measuring (24 ms given our frame rate of 41.7 Hz). So we are not lacking in temporal resolution.

But what about spatial resolution? Even though our pixels are 0.242 \(\mu m\) in size doesn’t mean that we can truly measure dynamics occurring around such small length scales.

The data used here was acquired using a not-so-great microscope with a 40x 0.5NA objective and a 0.3NA condenser. Assuming light of a wavelength of 550 nm, that means our theoretical resolution is \(\frac{1.22 \times 0.550}{0.5+0.3} = 0.763 \mu m\). Note that this is more than 3 times the pixel size.

As described in [Safari 2015], the maximum \(q\) is determined either by (1) the pixel size, (2) the smallest distance a particle can be detected moving between two frames given the frame rate, and (3) the smallest resolvable distance given the optical resolution limit. In the case of (3): \(q_{max} = \frac{2 \pi NA}{\lambda}\). For our data, that would be \(q_{max} = 5.7 \mu m^{-1}\).

So it seems we have found the culprit of the poor fitting at large :math:`q`: our limited optical resolution.

[Safari 2015] Safari, M. S., Vorontsova, M. A., Poling-Skutvik, R., Vekilov, P. G. & Conrad, J. C. Differential dynamic microscopy of weakly scattering and polydisperse protein-rich clusters. Phys. Rev. E 92, 042712 (2015).

[24]:
##################################################################################################
# Plotting the characteristic decay time tau vs wavevector q determined from the last three fits #
##################################################################################################

plt.figure(figsize=(10,10/1.618))
plt.loglog(fit01.q, fit01.parameters.loc['Tau'], 'ro', label="Not updating tau's initial guess")
plt.loglog(fit02.q, fit02.parameters.loc['Tau'], 'bo', label="Updating tau's initial guess")
plt.loglog(fit03.q, fit03.parameters.loc['Tau'], 'go', label="Restricting the stretching exponent")
plt.xlabel("$q$ ($\mu$m$^{-1})$")
plt.ylabel("Decay time, $\\tau$ (s)")
plt.axhline(y = 1.0/fit01.frame_rate)
plt.axvline(x = 5.7)
plt.legend(loc=0)
plt.title("Lines correspond roughly to limits of temporal (horizontal line) and spatial (vertical line) resolution")
[24]:
Text(0.5, 1.0, 'Lines correspond roughly to limits of temporal (horizontal line) and spatial (vertical line) resolution')
_images/Walkthrough_-_details_58_1.png

Determining the mean squared displacement from the ISF

If we assume that the particle displacements are Gaussian, then the ISF can be written as \(f(q,\Delta t) = \exp \left[ (-q^2 / 4) \left< \Delta r^2(\Delta t) \right> \right]\). We can then find the MSD, \(\left< \Delta r^2(\Delta t) \right>\) from the DDM matrix, \(D(q, \Delta t)\), using:

\[\left< \Delta r^2(\Delta t) \right> = \frac{4}{q^2} \ln \left[ \frac{A(q)}{A(q)-D(q,\Delta t)+B(q)} \right]\]

See equation 6 in [Bayles 2017] or equation 7 in [Edera 2017].

We find \(\left< \Delta r^2(\Delta t) \right>\) in this code with the method extract_MSD in the DDM_Fit class. The code will use the ‘Amplitude’ and ‘Background’ parameters of the fit. But, if those parameters are not present (as would be the case if we fit to the ISF instead of the DDM matrix) then the code will use the estimate for \(A(q)\) and \(B\) found using \(\left< | \tilde{I}(q, t) |^2 \right>_t\) as described previously in this document.

[Bayles 2017] Bayles, A. V., Squires, T. M. & Helgeson, M. E. Probe microrheology without particle tracking by differential dynamic microscopy. Rheol Acta 56, 863–869 (2017).

[Edera 2017] Edera, P., Bergamini, D., Trappe, V., Giavazzi, F. & Cerbino, R. Differential dynamic microscopy microrheology of soft materials: A tracking-free determination of the frequency-dependent loss and storage moduli. Phys. Rev. Materials 1, 073804 (2017).

[25]:
###############################################################################################
# Plotting the mean squared displacement as a function of lag time as determined from the ISF #
###############################################################################################

msd, msd_std = ddm_fit.extract_MSD(fit=fit01)
poly_fit_results = np.polyfit(msd.lagtime[1:10], msd[1:10], 1)

fig,ax = plt.subplots()
plt.loglog(msd.lagtime, msd, 'o', color='darkblue')
plt.plot(msd.lagtime, np.polyval(poly_fit_results,msd.lagtime), '-r')
plt.xlabel("Lag time (s)")
plt.ylabel("MSD ($\mu$m$^2$)")
plt.title("MSD vs lag time")

print("Diffusion coefficient: %.5f um^2/s" % (poly_fit_results[0]/4.))
C:\Users\rmcgorty\Anaconda3\lib\site-packages\xarray\core\computation.py:742: RuntimeWarning: invalid value encountered in log
  result_data = func(*input_data)
Diffusion coefficient: 0.66499 um^2/s
_images/Walkthrough_-_details_61_2.png
[ ]: