# Bulk Durable Delete

**URL:** <https://discuss.aerospike.com/t/bulk-durable-delete/8147>\
**Category:** How Developers Are Using Aerospike\
**Tags:** deletion, delete, durable-deletion\
**Created:** [January 14, 2021, 9:53am UTC](https://discuss.aerospike.com/t/bulk-durable-delete/8147 "2021-01-14T09:53:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fschaeffler](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/fschaeffler/32/1301_2.png) [@fschaeffler](https://discuss.aerospike.com/u/fschaeffler)\
**Post date:** [January 14, 2021, 9:53am UTC](https://discuss.aerospike.com/t/bulk-durable-delete/8147/1 "2021-01-14T09:53:02Z")

</div>

We’re using the official [Node.js Aerospike client](https://github.com/aerospike/aerospike-client-nodejs) and would like to make use of a bulk durable delete.

A non-persistent delete is doable via the following background UDF.

```auto
function delete(rec)
	aerospike:remove(rec)
end

```

A call to those UDFs is done via a background job like e.g.

```auto
const client = await Aerospike.connect()
const query = client.query(namespace, set, { nobins: true, filters: [filter] })
const job = await query.background(backgroundJob, 'delete')
await job.wait()

```

In case it’s done per-record and not via a stream, the functionality would look like the following one.

```auto
const client = await Aerospike.connect()
const query = client.query(namespace, set, { nobins: true, filters: [filter] })
const stream = query.foreach()

await new Promise((resolve, reject) => {
	stream.on('error', (error) => reject(error))
	stream.on('end', resolve)
	stream.on('data', ({ key }) => client.remove(key))
})

```

For a durable delete (Enterprise Feature), the only for us working implementation is the following one.

```auto
const client = await Aerospike.connect()
const query = client.query(namespace, set, { nobins: true, filters: [filter] })
const stream = query.foreach()

const policy = new policy.RemovePolicy({
	durableDelete: true
})

await new Promise((resolve, reject) => {
	stream.on('error', (error) => reject(error))
	stream.on('end', resolve)
	stream.on('data', ({ key }) => client.remove(key, policy))
})

```

As the provided solution for a durable delete is a per-record solution, the question is now **how to implement a bulk durable delete with a background UDF or if not possible via a stream-based solution**?

---

<div class="post-metadata">

**Author:** ![neelp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/neelp/32/1369_2.png) [@neelp](https://discuss.aerospike.com/u/neelp)\
**Post date:** [January 17, 2021, 5:49pm UTC](https://discuss.aerospike.com/t/bulk-durable-delete/8147/2 "2021-01-17T17:49:36Z")

</div>

You should be able to perform durable deletes with UDFs by setting durable-delete to true in the write policy. Alternatively you can perform a background scan with the delete operation and a filter expression to target specific records - again setting the durable-delete in write policy to true. Please let us know if this doesn’t work for some reason.

---

<div class="post-metadata">

**Author:** ![fschaeffler](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/fschaeffler/32/1301_2.png) [@fschaeffler](https://discuss.aerospike.com/u/fschaeffler)\
**Post date:** [January 18, 2021, 8:14am UTC](https://discuss.aerospike.com/t/bulk-durable-delete/8147/3 "2021-01-18T08:14:50Z")

</div>

Unfortunately, the documentation for the [background job](https://www.aerospike.com/apidocs/nodejs/Query.html#background__anchor) differs from the [sourcecode](https://github.com/aerospike/aerospike-client-nodejs/blob/7b4cd2fb35c4d9b3d269d394a7cff740a77f9024/src/main/commands/query_background.cc#L61).

The main difference is that the source code makes use of [WritePolicy](https://www.aerospike.com/apidocs/nodejs/WritePolicy.html), while the documentation makes use of [QueryPolicy](https://www.aerospike.com/apidocs/nodejs/QueryPolicy.html). This is actually quite some serious difference by means of functionality.

Another issue seems to be the performance of a durable delete via background UDF vs record-stream-remove-query. For our test set (~ 1 million records) the background UDF is 48% slower than the record-stream-remove-query.

To my understanding, the background UDF runs fully on the server without any client interaction or whatsoever. While the record-stream-remove-query needs to retrieve at least the key or digest from the server and then execute the actual remove-query.

**According to the server config documentation, [background-scan-max-rps](https://www.aerospike.com/docs/reference/configuration/#background-scan-max-rps) should be the right flag for performance tuning the background UDF, right?**

EDIT: Setting the server config flag of `background-scan-max-rps` to its maximum of `1000000` leads to test results where the background UDF is 59% faster than the `record-stream-remove-query`. So, yes, it is that flag 🙂

---

<div class="post-metadata">

**Author:** ![neelp](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/neelp/32/1369_2.png) [@neelp](https://discuss.aerospike.com/u/neelp)\
**Post date:** [January 19, 2021, 4:47pm UTC](https://discuss.aerospike.com/t/bulk-durable-delete/8147/4 "2021-01-19T16:47:02Z")

</div>

Great, you were able to get it working, and thanks for pointing out the doc inaccuracy (we will get it fixed). Indeed, UDFs are not usually to be used for highly performance sensitive tasks.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\
**Post date:** [April 13, 2021, 4:47pm UTC](https://discuss.aerospike.com/t/bulk-durable-delete/8147/5 "2021-04-13T16:47:03Z")

</div>

This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.